Nov 01 12:21:25.927303 ip-10-0-138-76 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 12:21:26.363560 ip-10-0-138-76 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:21:26.363560 ip-10-0-138-76 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 12:21:26.363560 ip-10-0-138-76 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:21:26.363560 ip-10-0-138-76 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:21:26.369087 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.368971    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:26.369087 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369083    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:26.369087 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369089    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:26.369087 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369095    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369105    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369110    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369112    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369115    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369118    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369121    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369124    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369126    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369129    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369131    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369134    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369137    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369139    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369142    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369145    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369147    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369150    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369152    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369155    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:26.369276 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369158    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369161    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369165    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369169    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369174    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369178    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369182    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369185    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369187    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369190    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369192    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369195    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369198    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369201    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369204    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369207    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369210    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369213    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369216    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369218    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:26.369752 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369221    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369223    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369226    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369228    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369231    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369234    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369236    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369239    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369241    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369247    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369251    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369255    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369259    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369263    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369265    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369268    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369272    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369277    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:26.370247 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369280    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369284    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369287    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369290    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369292    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369295    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369298    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369300    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369304    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369307    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369310    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369313    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369315    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369320    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369325    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369329    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369334    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369338    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369342    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:26.370667 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369344    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369347    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369350    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369361    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369363    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369366    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369368    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369371    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369374    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369377    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369380    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369383    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369386    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369389    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369391    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369394    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369397    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369400    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369403    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369408    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:26.371128 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369412    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369417    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369422    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369425    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369427    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.369431    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370448    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370460    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370465    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370470    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370474    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370477    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370479    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370482    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370485    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370488    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370501    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370504    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370507    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370510    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:26.371588 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370512    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370515    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370517    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370520    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370523    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370526    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370529    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370532    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370536    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370541    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370545    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370552    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370556    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370559    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370562    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370567    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370571    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370575    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370578    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:26.372163 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370581    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370583    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370586    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370589    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370591    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370594    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370596    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370599    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370601    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370604    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370613    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370616    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370621    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370624    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370629    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370633    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370636    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370639    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370642    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:26.372621 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370645    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370648    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370650    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370653    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370655    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370658    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370660    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370663    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370666    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370669    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370672    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370675    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370677    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370680    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370683    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370686    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370688    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370691    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370694    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:26.373080 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370697    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370701    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370705    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370709    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370717    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370725    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370728    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370731    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370733    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370736    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370738    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370741    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370744    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370747    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370749    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370751    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370754    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370757    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370760    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370762    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:26.373543 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370765    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370768    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370770    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370773    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370777    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370781    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370785    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370789    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370793    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370797    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370801    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370803    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370805    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370808    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.370810    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371404    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371413    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371416    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371419    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:26.374130 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371422    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371426    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371430    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371434    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371439    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371443    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371446    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371448    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371451    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371454    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371457    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371460    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371462    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371465    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371467    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371470    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371473    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371475    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371478    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371480    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:26.374566 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371483    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371485    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371488    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371490    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371493    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371495    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371498    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371501    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371503    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371508    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371512    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371518    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371522    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371526    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371529    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371531    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371534    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371536    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371539    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371542    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:26.375043 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371544    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371547    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371549    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371552    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371555    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371558    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371560    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371564    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371568    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371571    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371574    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371579    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371582    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371585    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371589    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371593    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371598    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371602    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371606    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:26.375529 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371608    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371610    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371614    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371617    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371619    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371623    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371625    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371628    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371630    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371633    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371636    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371638    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371641    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371643    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371646    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371649    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371651    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371653    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371656    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:26.375968 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371659    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371661    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371664    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371667    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371672    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371676    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371680    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371684    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371687    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371689    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371692    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371694    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371697    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371699    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371701    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371705    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371709    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371712    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371714    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371717    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:26.376424 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371720    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371722    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371725    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.371728    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372659    2582 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372671    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372721    2582 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372727    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372732    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372736    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372741    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372745    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372748    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372751    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372755    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372758    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372761    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372764    2582 flags.go:64] FLAG: --cgroup-root=""
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372767    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372771    2582 flags.go:64] FLAG: --client-ca-file=""
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372773    2582 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372776    2582 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372782    2582 flags.go:64] FLAG: --cluster-domain=""
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372785    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 12:21:26.376930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372789    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372792    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372795    2582 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372800    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372804    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372807    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372810    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372814    2582 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372817    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372820    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372824    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372827    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372831    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372834    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372837    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372841    2582 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372844    2582 flags.go:64] FLAG: --enable-server="true"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372847    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372851    2582 flags.go:64] FLAG: --event-burst="100"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372854    2582 flags.go:64] FLAG: --event-qps="50"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372857    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372860    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372863    2582 flags.go:64] FLAG: --eviction-hard=""
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372872    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 12:21:26.377540 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372875    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372878    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372881    2582 flags.go:64] FLAG: --eviction-soft=""
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372884    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372887    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372890    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372893    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372896    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372899    2582 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372902    2582 flags.go:64] FLAG: --feature-gates=""
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372906    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372909    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372913    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372915    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372919    2582 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372926    2582 flags.go:64] FLAG: --help="false"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372929    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372933    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372936    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372939    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372942    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372946    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372949    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 12:21:26.378147 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372952    2582 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372955    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372957    2582 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372961    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372964    2582 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372967    2582 flags.go:64] FLAG: --kube-reserved=""
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372970    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372973    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372976    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372979    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372982    2582 flags.go:64] FLAG: --lock-file=""
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372985    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372988    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372991    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372996    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.372999    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373002    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373005    2582 flags.go:64] FLAG: --logging-format="text"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373009    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373012    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373015    2582 flags.go:64] FLAG: --manifest-url=""
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373018    2582 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373022    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373025    2582 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373032    2582 flags.go:64] FLAG: --max-pods="110"
Nov 01 12:21:26.378687 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373035    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373038    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373042    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373045    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373061    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373067    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373071    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373079    2582 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373083    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373086    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373089    2582 flags.go:64] FLAG: --pod-cidr=""
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373092    2582 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373095    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373098    2582 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373101    2582 flags.go:64] FLAG: --port="10250"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373104    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373107    2582 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ef7d165ce2cbabee"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373110    2582 flags.go:64] FLAG: --qos-reserved=""
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373113    2582 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373116    2582 flags.go:64] FLAG: --register-node="true"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373119    2582 flags.go:64] FLAG: --register-with-taints=""
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373123    2582 flags.go:64] FLAG: --registry-burst="10"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373126    2582 flags.go:64] FLAG: --registry-qps="5"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373129    2582 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373132    2582 flags.go:64] FLAG: --reserved-memory=""
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373135    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 12:21:26.379288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373139    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373142    2582 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373145    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373148    2582 flags.go:64] FLAG: --runonce="false"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373150    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373155    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373158    2582 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373162    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373165    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373168    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373172    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373175    2582 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373178    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373181    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373184    2582 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373187    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373190    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373193    2582 flags.go:64] FLAG: --system-cgroups=""
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373196    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373202    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373204    2582 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373207    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373212    2582 flags.go:64] FLAG: --tls-min-version=""
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373215    2582 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373218    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 12:21:26.379975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373221    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373226    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373229    2582 flags.go:64] FLAG: --v="2"
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373233    2582 flags.go:64] FLAG: --version="false"
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373237    2582 flags.go:64] FLAG: --vmodule=""
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373246    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373250    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373393    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373397    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373400    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373403    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373405    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373409    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373412    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373415    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373418    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373420    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373423    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373426    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373429    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373431    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:26.380583 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373434    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373436    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373439    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373441    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373444    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373447    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373449    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373452    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373454    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373456    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373459    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373462    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373466    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373468    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373471    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373473    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373476    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373478    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373481    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373483    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:26.381086 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373486    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373488    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373491    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373496    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373499    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373502    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373505    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373507    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373510    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373513    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373516    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373519    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373521    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373524    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373527    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373529    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373531    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373534    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373536    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373539    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:26.381558 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373542    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373544    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373547    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373550    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373554    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373556    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373559    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373562    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373565    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373567    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373570    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373573    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373577    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373580    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373584    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373589    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373592    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373595    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373598    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:26.382060 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373600    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373603    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373606    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373609    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373611    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373614    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373617    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373619    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373622    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373625    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373627    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373630    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373632    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373635    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373637    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373640    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373642    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373647    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373649    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373652    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:26.382504 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373655    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373657    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373660    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373662    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373665    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373667    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373670    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373672    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373676    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373678    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373681    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373684    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373686    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.373714    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373751    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373757    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:26.382974 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373762    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373766    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373771    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373775    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373779    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373782    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373785    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373788    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373791    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373793    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373796    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373799    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373802    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373806    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373809    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373811    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373814    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373817    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373820    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:26.383368 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373823    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373832    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373836    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373838    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373841    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373844    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373848    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373851    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373853    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373856    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373859    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373861    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373864    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373867    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373871    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373874    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373877    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373880    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373882    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373885    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:26.383816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373887    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373890    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373893    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373895    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373898    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373900    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373904    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373907    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373909    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373912    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373914    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373917    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373920    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373922    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373925    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373927    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373930    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373933    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373938    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:26.384300 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373940    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373943    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373946    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373948    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373951    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373954    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373956    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373959    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373962    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373965    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373969    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373971    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373974    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373977    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373979    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373982    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373985    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373987    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373990    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:26.384759 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373993    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.373997    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374002    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374005    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374009    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374012    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374015    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374018    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374021    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374023    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374029    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374032    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374037    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374040    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374042    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374045    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374060    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374063    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374066    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:26.385223 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374068    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374071    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374074    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374078    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374082    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374085    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374087    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.374090    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.381109    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.381124    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.382703    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.382709    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382768    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382773    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382778    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382782    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382785    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382788    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382791    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382794    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:26.385700 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382797    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382799    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382802    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382805    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382807    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382810    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382813    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382815    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382818    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382820    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382823    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382826    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382829    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382831    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382834    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382838    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382840    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382843    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382846    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382849    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:26.386208 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382851    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382855    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382857    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382860    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382863    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382865    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382868    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382870    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382873    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382876    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382878    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382881    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382883    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382886    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382889    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382891    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382894    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382896    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382899    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382902    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:26.386740 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382905    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382907    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382920    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382923    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382926    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382929    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382931    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382934    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382937    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382940    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382943    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382945    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382948    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382951    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382954    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382956    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382959    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382962    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382964    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382967    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:26.387266 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382969    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382973    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382977    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382980    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382983    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382986    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382989    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382992    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382995    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.382999    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383002    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383005    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383008    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383011    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383013    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383016    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383019    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383021    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383024    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:26.387750 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383026    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383029    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383032    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383035    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383037    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383040    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383043    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383057    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383061    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383064    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383066    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383069    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383072    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383075    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383077    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383080    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383083    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383085    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:26.388209 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383088    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.383117    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383286    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383291    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383294    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383297    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383300    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383303    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383306    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383309    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383311    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383314    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383316    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383319    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383322    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:26.388644 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383324    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383327    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383330    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383333    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383336    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383339    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383342    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383344    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383347    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383349    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383352    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383355    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383358    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383360    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383363    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383366    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383368    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383371    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383374    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:26.389003 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383377    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383380    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383384    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383388    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383391    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383394    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383397    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383400    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383402    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383405    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383408    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383410    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383413    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383415    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383418    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383420    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383424    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383428    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383431    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:26.389502 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383434    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383436    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383439    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383441    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383444    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383446    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383449    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383451    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383454    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383456    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383459    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383461    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383464    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383466    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383469    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383472    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383474    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383477    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383479    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383482    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:26.389950 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383485    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383488    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383490    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383493    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383495    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383498    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383500    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383503    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383505    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383508    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383511    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383513    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383516    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383518    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383521    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383523    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383526    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383528    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383531    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:26.390438 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383533    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383536    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383538    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383540    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383543    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383546    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383548    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383551    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383554    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383557    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383559    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383562    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383564    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383567    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383569    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383572    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:26.390889 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.383595    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383626    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383630    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383633    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383636    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383638    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383641    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383644    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383647    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383649    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383652    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383655    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383657    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383660    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383663    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383666    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383669    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383671    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383674    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:26.391413 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383677    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383679    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383682    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383685    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383688    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383690    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383693    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383695    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383698    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383700    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383703    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383706    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383708    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383711    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383713    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383716    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383719    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383721    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383724    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:26.391856 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383726    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383728    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383732    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383736    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383739    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383742    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383744    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383747    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383750    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383753    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383755    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383760    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383762    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383765    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383768    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383770    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383775    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383778    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383781    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:26.392324 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383784    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383787    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383789    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383792    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383795    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383797    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383800    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383802    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383805    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383807    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383810    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383812    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383815    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383817    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383820    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383823    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383826    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383829    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383831    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383834    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:26.392760 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383837    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383840    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383842    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383845    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383848    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383851    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383854    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383856    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383859    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383861    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383864    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383866    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383869    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383871    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383874    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383876    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383879    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383881    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383883    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:26.393251 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383886    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383888    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383891    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383893    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383897    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383900    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383902    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383906    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383909    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383912    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:26.383914    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.383967    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.386270    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.387312    2582 server.go:1014] "Starting client certificate rotation"
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.387405    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.388211    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 12:21:26.393737 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.390168    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 12:21:26.395605 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.395586    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 12:21:26.409613 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.409592    2582 log.go:25] "Validated CRI v1 runtime API"
Nov 01 12:21:26.413404 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.413386    2582 log.go:25] "Validated CRI v1 image API"
Nov 01 12:21:26.414238 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.414214    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 12:21:26.417031 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.417013    2582 fs.go:135] Filesystem UUIDs: map[0ba2cb13-acb0-4790-9d5a-d8d737293e55:/dev/nvme0n1p3 30bfab0c-9e90-4341-a0e9-95f92e235026:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 01 12:21:26.417083 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.417031    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 12:21:26.423396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.423291    2582 manager.go:217] Machine: {Timestamp:2026-11-01 12:21:26.421841995 +0000 UTC m=+0.372848367 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3123923 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9afed2e728359a1ef0545f81cc33 SystemUUID:ec2a9afe-d2e7-2835-9a1e-f0545f81cc33 BootID:29ce00db-89e8-478c-9b29-af788b1b0981 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d3:c3:72:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d3:c3:72:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:b2:db:c9:fa:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 12:21:26.423396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.423395    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 12:21:26.423508 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.423481    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 12:21:26.427125 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.427102    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 12:21:26.427270 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.427127    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-76.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 12:21:26.427316 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.427272    2582 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 12:21:26.427316 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.427281    2582 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 12:21:26.427316 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.427307    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 12:21:26.427390 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.427324    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 12:21:26.428513 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.428503    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 12:21:26.432879 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.432867    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 12:21:26.433007 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.432992    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 12:21:26.433832 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.433818    2582 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 12:21:26.433864 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.433839    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 12:21:26.433864 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.433849    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 12:21:26.433864 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.433858    2582 kubelet.go:405] "Adding apiserver pod source"
Nov 01 12:21:26.433982 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.433866    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 12:21:26.434250 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.434235    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 12:21:26.436662 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.436640    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 12:21:26.443538 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.443525    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 12:21:26.444483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.444472    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 12:21:26.444518 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.444502    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 12:21:26.444656 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.444646    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 12:21:26.444685 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.444671    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 12:21:26.444685 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.444681    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 12:21:26.444751 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.444688    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 12:21:26.444751 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.444697    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 12:21:26.444751 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.444704    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 12:21:26.445392 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.445382    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 12:21:26.445426 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.445393    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 12:21:26.445426 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.445400    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 12:21:26.445426 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.445413    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 12:21:26.445426 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.445427    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 12:21:26.445545 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.445436    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 12:21:26.446178 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.446167    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 12:21:26.446178 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.446176    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 12:21:26.450018 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.450003    2582 server.go:1263] "Started kubelet"
Nov 01 12:21:26.450114 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.450072    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 12:21:26.450231 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.450170    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 12:21:26.450276 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.450262    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 12:21:26.450885 ip-10-0-138-76 systemd[1]: Started Kubernetes Kubelet.
Nov 01 12:21:26.451948 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.451928    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 12:21:26.452680 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.452664    2582 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 12:21:26.457590 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.457558    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 12:21:26.459031 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.459011    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 12:21:26.459031 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.459021    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 12:21:26.459175 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.459121    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-76.us-west-2.compute.internal\" not found"
Nov 01 12:21:26.459781 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.459752    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 12:21:26.459872 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.459850    2582 factory.go:55] Registering systemd factory
Nov 01 12:21:26.459872 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.459863    2582 factory.go:223] Registration of the systemd container factory successfully
Nov 01 12:21:26.459980 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.459914    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 12:21:26.459980 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.459930    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 12:21:26.460120 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.460102    2582 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 12:21:26.460120 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.460116    2582 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 12:21:26.460120 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.460102    2582 factory.go:153] Registering CRI-O factory
Nov 01 12:21:26.460355 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.460137    2582 factory.go:223] Registration of the crio container factory successfully
Nov 01 12:21:26.460355 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.460188    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 12:21:26.460355 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.460208    2582 factory.go:103] Registering Raw factory
Nov 01 12:21:26.460355 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.460224    2582 manager.go:1199] Started watching for new ooms in manager
Nov 01 12:21:26.461313 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.461300    2582 manager.go:319] Starting recovery of all containers
Nov 01 12:21:26.471262 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.471241    2582 manager.go:324] Recovery completed
Nov 01 12:21:26.475686 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.475669    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:26.477358 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.477338    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:26.477425 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.477378    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:26.477425 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.477396    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:26.478891 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.477987    2582 cpu_manager.go:226] "Starting" policy="none"
Nov 01 12:21:26.478891 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.478002    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 12:21:26.478891 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.478018    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 12:21:26.478891 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.478003    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:26.479201 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.479124    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:26.479201 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.479152    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:26.479201 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.479164    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:26.559650 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.559618    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-76.us-west-2.compute.internal\" not found"
Nov 01 12:21:26.585671 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.585639    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-76.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 12:21:26.585671 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.585674    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-76.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 12:21:26.588003 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.585528    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-76.us-west-2.compute.internal.18e3eb344ad5dc76  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-76.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-76.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 12:21:26.449945718 +0000 UTC m=+0.400952090,LastTimestamp:2026-11-01 12:21:26.449945718 +0000 UTC m=+0.400952090,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-76.us-west-2.compute.internal,}"
Nov 01 12:21:26.621141 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.621083    2582 policy_none.go:50] "Start"
Nov 01 12:21:26.621141 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.621102    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 12:21:26.621141 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.621122    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 12:21:26.623009 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.622994    2582 policy_none.go:44] "Start"
Nov 01 12:21:26.650706 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.645124    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 12:21:26.650706 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.646395    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 12:21:26.650706 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.646412    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 12:21:26.650706 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.646435    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 12:21:26.650706 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.646475    2582 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 12:21:26.658638 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.658604    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 12:21:26.659899 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.659883    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-76.us-west-2.compute.internal\" not found"
Nov 01 12:21:26.660993 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.660982    2582 manager.go:341] "Starting Device Plugin manager"
Nov 01 12:21:26.661752 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.661738    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 12:21:26.661795 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.661753    2582 server.go:87] "Starting device plugin registration server"
Nov 01 12:21:26.661974 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.661963    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 12:21:26.662008 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.661976    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 12:21:26.662099 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.662078    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 12:21:26.662403 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.662156    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 12:21:26.662403 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.662166    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 12:21:26.662660 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.662645    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 12:21:26.662725 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.662682    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-76.us-west-2.compute.internal\" not found"
Nov 01 12:21:26.665910 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.665894    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 12:21:26.665984 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.665951    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hcg5f"
Nov 01 12:21:26.745735 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.745712    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:26.746790 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.746773    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"]
Nov 01 12:21:26.746839 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.746828    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:26.747432 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.747417    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:26.747656 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.747643    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:26.747700 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.747676    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:26.747700 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.747688    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:26.749923 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.749911    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:26.750119 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.750082    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.750199 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.750123    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:26.750654 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.750639    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:26.750714 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.750668    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:26.750714 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.750678    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:26.750790 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.750641    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:26.750790 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.750736    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:26.750790 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.750747    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:26.752739 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.752720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.752822 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.752749    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:26.753399 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.753382    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:26.753470 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.753407    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:26.753470 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.753422    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:26.761725 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.761705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d08d3cfd44df29731faafa69be6f8df6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"d08d3cfd44df29731faafa69be6f8df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.761806 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.761731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.761806 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.761751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.762743 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.762729    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:26.764184 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.764171    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:26.764228 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.764200    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:26.764228 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.764210    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:26.764290 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.764241    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.783672 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.783521    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-76.us-west-2.compute.internal\" not found" node="ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.786155 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.786138    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-76.us-west-2.compute.internal\" not found" node="ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.847953 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.847923    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:26.849205 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.849187    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:26.849312 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.849293    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.849362 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:26.849325    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-76.us-west-2.compute.internal\": node \"ip-10-0-138-76.us-west-2.compute.internal\" not found"
Nov 01 12:21:26.860044 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.860023    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.861899 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.861879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.861986 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.861906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.861986 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.861923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d08d3cfd44df29731faafa69be6f8df6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"d08d3cfd44df29731faafa69be6f8df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.861986 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.861949    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.861986 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.861962    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d08d3cfd44df29731faafa69be6f8df6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"d08d3cfd44df29731faafa69be6f8df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.861986 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.861969    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:26.937578 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.937523    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:21:26.937674 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:26.937611    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:27.034353 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.034324    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:21:27.088557 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.088530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:27.091723 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.091708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:21:27.387070 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.386980    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 12:21:27.387585 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.387115    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 12:21:27.387585 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.387118    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 12:21:27.387585 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.387141    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 12:21:27.434772 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.434743    2582 apiserver.go:52] "Watching apiserver"
Nov 01 12:21:27.459858 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.459828    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 12:21:27.654613 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.654538    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 12:21:27.654862 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.654843    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-fpq52","openshift-ovn-kubernetes/ovnkube-node-vrjcf","kube-system/global-pull-secret-syncer-v7lhr","kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal","openshift-multus/multus-lxjxw","openshift-network-operator/iptables-alerter-xwv48","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5","openshift-dns/node-resolver-kfdrq","openshift-multus/network-metrics-daemon-p779s","openshift-network-diagnostics/network-check-target-6zjj4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-tg2vq","kube-system/konnectivity-agent-bm5xm","openshift-cluster-node-tuning-operator/tuned-dvz6d"]
Nov 01 12:21:27.657291 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.657271    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.659342 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.659325    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.661283 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.661265    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:27.661344 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.661319    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:27.663123 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.663108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.665027 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665011    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-kubelet-config\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:27.665095 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8b43ebe-6a94-45f6-a613-0931f4607e9e-serviceca\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.665126 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665109    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.665153 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-systemd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665203 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-netd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665281 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665243    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-var-lib-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665346 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-script-lib\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665395 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-slash\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665395 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-netns\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665460 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-log-socket\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665460 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-env-overrides\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665517 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovn-node-metrics-cert\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665517 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:27.665568 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-systemd-units\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665568 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-etc-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665621 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-ovn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665621 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665688 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665688 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-dbus\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:27.665784 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-bin\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665784 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-config\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665784 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665746    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8b43ebe-6a94-45f6-a613-0931f4607e9e-host\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.665784 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-kubelet\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665896 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665896 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-node-log\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.665896 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t59b\" (UniqueName: \"kubernetes.io/projected/f8b43ebe-6a94-45f6-a613-0931f4607e9e-kube-api-access-4t59b\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.665896 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.665888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rtn\" (UniqueName: \"kubernetes.io/projected/19e0e140-91e4-4d92-b1ac-0ea2115e3467-kube-api-access-l4rtn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.667060 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.667030    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.668998 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.668981    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.670901 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.670867    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 12:16:26 +0000 UTC" deadline="2028-06-23 00:02:06.879597565 +0000 UTC"
Nov 01 12:21:27.670901 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.670899    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14387h40m39.208701662s"
Nov 01 12:21:27.671033 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.670952    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:27.671033 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.670996    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:27.672931 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.672917    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:27.673013 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.672961    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:27.675076 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.675061    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.677359 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.677346    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:27.679281 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.679267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.702394 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.702378    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 12:21:27.702493 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.702475    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 12:21:27.702535 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.702475    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 12:21:27.702865 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.702854    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zhblh\""
Nov 01 12:21:27.704020 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.704004    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 12:21:27.704098 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.704087    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 12:21:27.704137 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.704109    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2z6g\""
Nov 01 12:21:27.704166 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.704156    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 12:21:27.704225 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.704213    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 12:21:27.704296 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.704286    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 12:21:27.704354 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.704344    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 12:21:27.708151 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.708131    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 12:21:27.708229 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.708152    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 12:21:27.708283 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.708269    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 12:21:27.708591 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.708573    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 12:21:27.708684 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.708632    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z9pcl\""
Nov 01 12:21:27.720242 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.720222    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:21:27.720349 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.720250    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qrhfv\""
Nov 01 12:21:27.742832 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.742814    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 12:21:27.742832 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.742823    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 12:21:27.742951 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.742869    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 12:21:27.742951 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.742871    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 12:21:27.742951 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.742900    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nsq77\""
Nov 01 12:21:27.745689 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.745678    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 12:21:27.745827 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.745815    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 12:21:27.745872 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.745815    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sfjvd\""
Nov 01 12:21:27.745872 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.745846    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 12:21:27.745872 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.745858    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 12:21:27.747724 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.747712    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhf7p\""
Nov 01 12:21:27.747804 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.747794    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 12:21:27.747855 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.747840    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7twmh\""
Nov 01 12:21:27.747855 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.747850    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 12:21:27.747924 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.747866    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:21:27.747924 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.747872    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 12:21:27.747924 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.747898    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 12:21:27.748063 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.747974    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9cl4q\""
Nov 01 12:21:27.748063 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.748003    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 12:21:27.748142 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.748067    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8gxq8"
Nov 01 12:21:27.760535 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.760523    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 12:21:27.766018 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.765999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8b43ebe-6a94-45f6-a613-0931f4607e9e-serviceca\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.766123 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-systemd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766123 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-netd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766123 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766075    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-system-cni-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.766123 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-binary-copy\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.766123 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766101    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-systemd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766123 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-var-lib-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766172    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-netd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-socket-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-registration-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-cnibin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766237    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-var-lib-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bmhd\" (UniqueName: \"kubernetes.io/projected/5cedf131-a296-4792-8328-f3ad030d3003-kube-api-access-7bmhd\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwsm4\" (UniqueName: \"kubernetes.io/projected/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-kube-api-access-zwsm4\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-script-lib\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5291295-5fd1-4e6e-b52b-c940c088168d-konnectivity-ca\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-netns\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-log-socket\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-env-overrides\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovn-node-metrics-cert\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766610    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-log-socket\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-iptables-alerter-script\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766573    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-netns\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5291295-5fd1-4e6e-b52b-c940c088168d-agent-certs\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-multus\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-etc-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-ovn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766760    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-k8s-cni-cncf-io\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c09c0770-6cec-4827-9fb0-b07464653b27-tmp-dir\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.766857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-var-lib-kubelet\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-host\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkmg\" (UniqueName: \"kubernetes.io/projected/c09c0770-6cec-4827-9fb0-b07464653b27-kube-api-access-fkkmg\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-run\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766944    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-lib-modules\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-bin\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.766989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-config\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767019    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9jg\" (UniqueName: \"kubernetes.io/projected/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-kube-api-access-zb9jg\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767035    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8b43ebe-6a94-45f6-a613-0931f4607e9e-serviceca\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-cni-binary-copy\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767142    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-env-overrides\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-kubernetes\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767175    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-bin\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-kubelet\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.767483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767231    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-hostroot\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rtn\" (UniqueName: \"kubernetes.io/projected/19e0e140-91e4-4d92-b1ac-0ea2115e3467-kube-api-access-l4rtn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767269    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-script-lib\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-kubelet-config\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767273    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-etc-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-device-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767320    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-ovn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767340    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767348    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-kubelet-config\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767389    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-kubelet\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767587    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlch9\" (UniqueName: \"kubernetes.io/projected/46433966-ea95-45b6-9092-c2f73bd7cd53-kube-api-access-qlch9\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767585    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767615    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-system-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.768236 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-multus-daemon-config\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysconfig\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-sys\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767744    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-tuned\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-netns\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767765    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-config\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c09c0770-6cec-4827-9fb0-b07464653b27-hosts-file\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gs79\" (UniqueName: \"kubernetes.io/projected/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-kube-api-access-5gs79\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-systemd\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-slash\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767921    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-slash\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-bin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-kubelet\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.767997    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-conf-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768011    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-conf\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-systemd-units\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.768734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-socket-dir-parent\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768144    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-systemd-units\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-etc-kubernetes\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cnibin\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.768233    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.768294    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret podName:370a3bfd-1661-4e44-b5d5-5f6b83db1f6c nodeName:}" failed. No retries permitted until 2026-11-01 12:21:28.26826653 +0000 UTC m=+2.219272901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret") pod "global-pull-secret-syncer-v7lhr" (UID: "370a3bfd-1661-4e44-b5d5-5f6b83db1f6c") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768301    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-dbus\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-host-slash\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqbs7\" (UniqueName: \"kubernetes.io/projected/3888b182-8fb4-49bd-9af3-873bcc3efa92-kube-api-access-cqbs7\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-modprobe-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768374    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-dbus\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-multus-certs\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768416    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-tmp\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8b43ebe-6a94-45f6-a613-0931f4607e9e-host\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.769175 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-node-log\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.769586 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t59b\" (UniqueName: \"kubernetes.io/projected/f8b43ebe-6a94-45f6-a613-0931f4607e9e-kube-api-access-4t59b\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.769586 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768498    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-node-log\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.769586 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768509    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8b43ebe-6a94-45f6-a613-0931f4607e9e-host\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.769586 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.768536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-sys-fs\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.770642 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.770629    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovn-node-metrics-cert\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.816235 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.816210    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rtn\" (UniqueName: \"kubernetes.io/projected/19e0e140-91e4-4d92-b1ac-0ea2115e3467-kube-api-access-l4rtn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:27.816760 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.816746    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t59b\" (UniqueName: \"kubernetes.io/projected/f8b43ebe-6a94-45f6-a613-0931f4607e9e-kube-api-access-4t59b\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:27.869529 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5291295-5fd1-4e6e-b52b-c940c088168d-konnectivity-ca\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:27.869624 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.869624 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-iptables-alerter-script\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.869624 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5291295-5fd1-4e6e-b52b-c940c088168d-agent-certs\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:27.869756 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869639    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-multus\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.869756 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-k8s-cni-cncf-io\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.869756 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c09c0770-6cec-4827-9fb0-b07464653b27-tmp-dir\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.869756 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869707    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.869756 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869732    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-multus\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.869910 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869757    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-k8s-cni-cncf-io\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.869910 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.869910 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-var-lib-kubelet\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.869910 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-host\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.869910 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkmg\" (UniqueName: \"kubernetes.io/projected/c09c0770-6cec-4827-9fb0-b07464653b27-kube-api-access-fkkmg\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869933    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-var-lib-kubelet\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869962    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-run\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-lib-modules\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870007    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-host\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.869961    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9jg\" (UniqueName: \"kubernetes.io/projected/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-kube-api-access-zb9jg\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-cni-binary-copy\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870098    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c09c0770-6cec-4827-9fb0-b07464653b27-tmp-dir\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870111    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-iptables-alerter-script\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870132    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-lib-modules\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.870159 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870149    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5291295-5fd1-4e6e-b52b-c940c088168d-konnectivity-ca\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-kubernetes\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870196    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-run\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-hostroot\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-device-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870260    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-kubernetes\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870293    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870325    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-hostroot\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870384    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-device-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870429    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.870573 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870554    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-cni-binary-copy\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.870939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlch9\" (UniqueName: \"kubernetes.io/projected/46433966-ea95-45b6-9092-c2f73bd7cd53-kube-api-access-qlch9\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.870939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870818    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-system-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.870939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.870939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870856    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.870939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-multus-daemon-config\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.870939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysconfig\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-sys\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-tuned\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.870939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870928    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-system-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-netns\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870932    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c09c0770-6cec-4827-9fb0-b07464653b27-hosts-file\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870979    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysconfig\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gs79\" (UniqueName: \"kubernetes.io/projected/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-kube-api-access-5gs79\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.870981    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-sys\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871003    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-netns\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-systemd\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-bin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-kubelet\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871094    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-systemd\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-conf-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871143    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-kubelet\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871098    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-bin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871145    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-conf-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871039    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c09c0770-6cec-4827-9fb0-b07464653b27-hosts-file\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-conf\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.871323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-socket-dir-parent\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871289    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-conf\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-etc-kubernetes\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cnibin\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871346    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-socket-dir-parent\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-host-slash\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871389    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-etc-kubernetes\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871404    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cnibin\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871410    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-host-slash\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbs7\" (UniqueName: \"kubernetes.io/projected/3888b182-8fb4-49bd-9af3-873bcc3efa92-kube-api-access-cqbs7\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-modprobe-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871549    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-modprobe-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-multus-certs\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-tmp\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-sys-fs\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871647    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-multus-certs\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.871929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-system-cni-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-binary-copy\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871721    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-system-cni-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871747    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-sys-fs\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-socket-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-registration-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-cnibin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmhd\" (UniqueName: \"kubernetes.io/projected/5cedf131-a296-4792-8328-f3ad030d3003-kube-api-access-7bmhd\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwsm4\" (UniqueName: \"kubernetes.io/projected/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-kube-api-access-zwsm4\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871957    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-socket-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.871967    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-cnibin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.872033    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-multus-daemon-config\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.872043    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-registration-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.872246    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.872297    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs podName:3888b182-8fb4-49bd-9af3-873bcc3efa92 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:28.372281057 +0000 UTC m=+2.323287428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs") pod "network-metrics-daemon-p779s" (UID: "3888b182-8fb4-49bd-9af3-873bcc3efa92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:27.872498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.872383    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5291295-5fd1-4e6e-b52b-c940c088168d-agent-certs\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:27.872914 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.872716    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-binary-copy\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.872914 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.872747    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.873624 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.873608    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-tuned\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.873739 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.873724    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-tmp\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.917983 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.917920    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:27.917983 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.917940    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:27.917983 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.917950    2582 projected.go:196] Error preparing data for projected volume kube-api-access-x26dv for pod openshift-network-diagnostics/network-check-target-6zjj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:27.918226 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:27.918040    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv podName:47f53f35-b29e-4db5-91c8-0df05af362e7 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:28.418013595 +0000 UTC m=+2.369019966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x26dv" (UniqueName: "kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv") pod "network-check-target-6zjj4" (UID: "47f53f35-b29e-4db5-91c8-0df05af362e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:27.919625 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.919599    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9jg\" (UniqueName: \"kubernetes.io/projected/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-kube-api-access-zb9jg\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:27.919948 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.919930    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkmg\" (UniqueName: \"kubernetes.io/projected/c09c0770-6cec-4827-9fb0-b07464653b27-kube-api-access-fkkmg\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:27.920245 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.920224    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlch9\" (UniqueName: \"kubernetes.io/projected/46433966-ea95-45b6-9092-c2f73bd7cd53-kube-api-access-qlch9\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:27.920543 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.920525    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbs7\" (UniqueName: \"kubernetes.io/projected/3888b182-8fb4-49bd-9af3-873bcc3efa92-kube-api-access-cqbs7\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:27.920870 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.920851    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwsm4\" (UniqueName: \"kubernetes.io/projected/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-kube-api-access-zwsm4\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:27.920931 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.920919    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmhd\" (UniqueName: \"kubernetes.io/projected/5cedf131-a296-4792-8328-f3ad030d3003-kube-api-access-7bmhd\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:27.921535 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.921519    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gs79\" (UniqueName: \"kubernetes.io/projected/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-kube-api-access-5gs79\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:27.976981 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:27.976947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:21:28.003349 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.003328    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxjxw"
Nov 01 12:21:28.009803 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.009787    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:28.026169 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.026153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:21:28.040600 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.040582    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:21:28.053933 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.053919    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:21:28.077740 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.077719    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:21:28.087193 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.087176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:28.090450 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.090434    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:21:28.275907 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.275838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:28.276142 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:28.275981    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:28.276142 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:28.276045    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret podName:370a3bfd-1661-4e44-b5d5-5f6b83db1f6c nodeName:}" failed. No retries permitted until 2026-11-01 12:21:29.27602378 +0000 UTC m=+3.227030149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret") pod "global-pull-secret-syncer-v7lhr" (UID: "370a3bfd-1661-4e44-b5d5-5f6b83db1f6c") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:28.292379 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:28.292321    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839bae4638948a932829e431666671db.slice/crio-a024a7bf0ef503a9b7d0b9203af8167c404ea1e5cbcba3cd270ba730a79a3d14 WatchSource:0}: Error finding container a024a7bf0ef503a9b7d0b9203af8167c404ea1e5cbcba3cd270ba730a79a3d14: Status 404 returned error can't find the container with id a024a7bf0ef503a9b7d0b9203af8167c404ea1e5cbcba3cd270ba730a79a3d14
Nov 01 12:21:28.292730 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:28.292710    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08d3cfd44df29731faafa69be6f8df6.slice/crio-36ea08297cbe07593735aaf7420137f4a1f2f8e85422bda88076bd722c990946 WatchSource:0}: Error finding container 36ea08297cbe07593735aaf7420137f4a1f2f8e85422bda88076bd722c990946: Status 404 returned error can't find the container with id 36ea08297cbe07593735aaf7420137f4a1f2f8e85422bda88076bd722c990946
Nov 01 12:21:28.298241 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.298209    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:21:28.326070 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.326026    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:28.376489 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.376467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:28.376608 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:28.376592    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:28.376650 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:28.376643    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs podName:3888b182-8fb4-49bd-9af3-873bcc3efa92 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:29.376628725 +0000 UTC m=+3.327635086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs") pod "network-metrics-daemon-p779s" (UID: "3888b182-8fb4-49bd-9af3-873bcc3efa92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:28.477742 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.477712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:28.478213 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:28.477850    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:28.478213 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:28.477867    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:28.478213 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:28.477877    2582 projected.go:196] Error preparing data for projected volume kube-api-access-x26dv for pod openshift-network-diagnostics/network-check-target-6zjj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:28.478213 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:28.477919    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv podName:47f53f35-b29e-4db5-91c8-0df05af362e7 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:29.477906821 +0000 UTC m=+3.428913179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x26dv" (UniqueName: "kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv") pod "network-check-target-6zjj4" (UID: "47f53f35-b29e-4db5-91c8-0df05af362e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:28.650717 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.650623    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"839bae4638948a932829e431666671db","Type":"ContainerStarted","Data":"a024a7bf0ef503a9b7d0b9203af8167c404ea1e5cbcba3cd270ba730a79a3d14"}
Nov 01 12:21:28.651653 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.651632    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"d08d3cfd44df29731faafa69be6f8df6","Type":"ContainerStarted","Data":"36ea08297cbe07593735aaf7420137f4a1f2f8e85422bda88076bd722c990946"}
Nov 01 12:21:28.750859 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.750810    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 12:16:27 +0000 UTC" deadline="2028-07-13 04:08:10.798926992 +0000 UTC"
Nov 01 12:21:28.750859 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.750848    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14871h46m42.048081554s"
Nov 01 12:21:28.788916 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:28.788895    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e0e140_91e4_4d92_b1ac_0ea2115e3467.slice/crio-80e025ea2ea4cacbf74cce32958967710c1fd4d417b205cb290b96e28946efc1 WatchSource:0}: Error finding container 80e025ea2ea4cacbf74cce32958967710c1fd4d417b205cb290b96e28946efc1: Status 404 returned error can't find the container with id 80e025ea2ea4cacbf74cce32958967710c1fd4d417b205cb290b96e28946efc1
Nov 01 12:21:28.833079 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:28.833025    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cedf131_a296_4792_8328_f3ad030d3003.slice/crio-5fd19072fe78d3948dfeddfb760348f6ba8d08f7ac323b3839980b06cf5884f5 WatchSource:0}: Error finding container 5fd19072fe78d3948dfeddfb760348f6ba8d08f7ac323b3839980b06cf5884f5: Status 404 returned error can't find the container with id 5fd19072fe78d3948dfeddfb760348f6ba8d08f7ac323b3839980b06cf5884f5
Nov 01 12:21:28.855816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:28.855788    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffad4d99_42c6_4151_a4cf_e22b3ecd03e2.slice/crio-1ec4e303cde0fe00ebe4cf59d0ff5ef5eaa4808adeda24c467573b2b44bbe584 WatchSource:0}: Error finding container 1ec4e303cde0fe00ebe4cf59d0ff5ef5eaa4808adeda24c467573b2b44bbe584: Status 404 returned error can't find the container with id 1ec4e303cde0fe00ebe4cf59d0ff5ef5eaa4808adeda24c467573b2b44bbe584
Nov 01 12:21:28.872899 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:28.872874    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693b0401_a9f9_41a4_b2e8_d3e3d6f37477.slice/crio-81fffd4e6988fb157febb30c1ca3844c26047f5cecb478327b88b6eb667108da WatchSource:0}: Error finding container 81fffd4e6988fb157febb30c1ca3844c26047f5cecb478327b88b6eb667108da: Status 404 returned error can't find the container with id 81fffd4e6988fb157febb30c1ca3844c26047f5cecb478327b88b6eb667108da
Nov 01 12:21:28.903580 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:28.903556    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5291295_5fd1_4e6e_b52b_c940c088168d.slice/crio-0215e6a2e22be96d6ed38a4f26830d0cf4fa8ec7763300c0e4b01dcdf876e710 WatchSource:0}: Error finding container 0215e6a2e22be96d6ed38a4f26830d0cf4fa8ec7763300c0e4b01dcdf876e710: Status 404 returned error can't find the container with id 0215e6a2e22be96d6ed38a4f26830d0cf4fa8ec7763300c0e4b01dcdf876e710
Nov 01 12:21:28.903967 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:28.903950    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46433966_ea95_45b6_9092_c2f73bd7cd53.slice/crio-16839b5525d977e2ceb2cf35c97f94ddccca438d46194fac581c29d9c5798dab WatchSource:0}: Error finding container 16839b5525d977e2ceb2cf35c97f94ddccca438d46194fac581c29d9c5798dab: Status 404 returned error can't find the container with id 16839b5525d977e2ceb2cf35c97f94ddccca438d46194fac581c29d9c5798dab
Nov 01 12:21:28.926661 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:28.926640    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09c0770_6cec_4827_9fb0_b07464653b27.slice/crio-d176df9d439e171136a90a031eccdffbe8063a47d766bf548753b7e098f396fc WatchSource:0}: Error finding container d176df9d439e171136a90a031eccdffbe8063a47d766bf548753b7e098f396fc: Status 404 returned error can't find the container with id d176df9d439e171136a90a031eccdffbe8063a47d766bf548753b7e098f396fc
Nov 01 12:21:28.931861 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.929904    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:28.935398 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:28.935381    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:29.013972 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:29.013944    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a91a46a_94a6_466a_8c8f_8ae2f5ff4d36.slice/crio-81d22b5069588bfda90aa35e3a58cfdd69ee843b07abe92fd22ef6ee3dd9a0de WatchSource:0}: Error finding container 81d22b5069588bfda90aa35e3a58cfdd69ee843b07abe92fd22ef6ee3dd9a0de: Status 404 returned error can't find the container with id 81d22b5069588bfda90aa35e3a58cfdd69ee843b07abe92fd22ef6ee3dd9a0de
Nov 01 12:21:29.285093 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.285009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:29.285220 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.285155    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:29.285220 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.285216    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret podName:370a3bfd-1661-4e44-b5d5-5f6b83db1f6c nodeName:}" failed. No retries permitted until 2026-11-01 12:21:31.285198974 +0000 UTC m=+5.236205349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret") pod "global-pull-secret-syncer-v7lhr" (UID: "370a3bfd-1661-4e44-b5d5-5f6b83db1f6c") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:29.385783 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.385751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:29.385925 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.385889    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:29.385974 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.385949    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs podName:3888b182-8fb4-49bd-9af3-873bcc3efa92 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:31.385935535 +0000 UTC m=+5.336941891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs") pod "network-metrics-daemon-p779s" (UID: "3888b182-8fb4-49bd-9af3-873bcc3efa92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:29.487026 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.486989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:29.487584 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.487194    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:29.487584 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.487220    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:29.487584 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.487232    2582 projected.go:196] Error preparing data for projected volume kube-api-access-x26dv for pod openshift-network-diagnostics/network-check-target-6zjj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:29.487584 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.487286    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv podName:47f53f35-b29e-4db5-91c8-0df05af362e7 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:31.487270607 +0000 UTC m=+5.438276976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x26dv" (UniqueName: "kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv") pod "network-check-target-6zjj4" (UID: "47f53f35-b29e-4db5-91c8-0df05af362e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:29.647322 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.646839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:29.647322 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.646871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:29.647322 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.646974    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:29.647322 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.646986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:29.647322 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.647096    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:29.647322 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:29.647171    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:29.654384 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.654356    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5" event={"ID":"46433966-ea95-45b6-9092-c2f73bd7cd53","Type":"ContainerStarted","Data":"16839b5525d977e2ceb2cf35c97f94ddccca438d46194fac581c29d9c5798dab"}
Nov 01 12:21:29.655692 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.655670    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bm5xm" event={"ID":"e5291295-5fd1-4e6e-b52b-c940c088168d","Type":"ContainerStarted","Data":"0215e6a2e22be96d6ed38a4f26830d0cf4fa8ec7763300c0e4b01dcdf876e710"}
Nov 01 12:21:29.657160 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.657139    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" event={"ID":"693b0401-a9f9-41a4-b2e8-d3e3d6f37477","Type":"ContainerStarted","Data":"81fffd4e6988fb157febb30c1ca3844c26047f5cecb478327b88b6eb667108da"}
Nov 01 12:21:29.658271 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.658244    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xwv48" event={"ID":"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2","Type":"ContainerStarted","Data":"1ec4e303cde0fe00ebe4cf59d0ff5ef5eaa4808adeda24c467573b2b44bbe584"}
Nov 01 12:21:29.659566 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.659548    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxjxw" event={"ID":"5cedf131-a296-4792-8328-f3ad030d3003","Type":"ContainerStarted","Data":"5fd19072fe78d3948dfeddfb760348f6ba8d08f7ac323b3839980b06cf5884f5"}
Nov 01 12:21:29.661130 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.661101    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" event={"ID":"19e0e140-91e4-4d92-b1ac-0ea2115e3467","Type":"ContainerStarted","Data":"80e025ea2ea4cacbf74cce32958967710c1fd4d417b205cb290b96e28946efc1"}
Nov 01 12:21:29.663477 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.663453    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dvz6d" event={"ID":"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36","Type":"ContainerStarted","Data":"81d22b5069588bfda90aa35e3a58cfdd69ee843b07abe92fd22ef6ee3dd9a0de"}
Nov 01 12:21:29.664638 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.664620    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kfdrq" event={"ID":"c09c0770-6cec-4827-9fb0-b07464653b27","Type":"ContainerStarted","Data":"d176df9d439e171136a90a031eccdffbe8063a47d766bf548753b7e098f396fc"}
Nov 01 12:21:29.666201 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.666180    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fpq52" event={"ID":"f8b43ebe-6a94-45f6-a613-0931f4607e9e","Type":"ContainerStarted","Data":"6c9208b0b436be94df86d2bb4a955ecb04f0ac2ad90da7afa1e23e96b3d1e459"}
Nov 01 12:21:29.751587 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.751523    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 12:16:27 +0000 UTC" deadline="2028-08-13 09:36:21.132137272 +0000 UTC"
Nov 01 12:21:29.751587 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:29.751576    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15621h14m51.380564797s"
Nov 01 12:21:31.302492 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:31.301804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:31.302492 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.301969    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:31.302492 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.302032    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret podName:370a3bfd-1661-4e44-b5d5-5f6b83db1f6c nodeName:}" failed. No retries permitted until 2026-11-01 12:21:35.30201371 +0000 UTC m=+9.253020085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret") pod "global-pull-secret-syncer-v7lhr" (UID: "370a3bfd-1661-4e44-b5d5-5f6b83db1f6c") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:31.402358 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:31.402323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:31.402580 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.402527    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:31.402656 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.402589    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs podName:3888b182-8fb4-49bd-9af3-873bcc3efa92 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:35.402571305 +0000 UTC m=+9.353577665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs") pod "network-metrics-daemon-p779s" (UID: "3888b182-8fb4-49bd-9af3-873bcc3efa92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:31.503239 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:31.503184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:31.503403 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.503349    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:31.503403 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.503364    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:31.503403 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.503375    2582 projected.go:196] Error preparing data for projected volume kube-api-access-x26dv for pod openshift-network-diagnostics/network-check-target-6zjj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:31.503578 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.503420    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv podName:47f53f35-b29e-4db5-91c8-0df05af362e7 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:35.503405996 +0000 UTC m=+9.454412355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x26dv" (UniqueName: "kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv") pod "network-check-target-6zjj4" (UID: "47f53f35-b29e-4db5-91c8-0df05af362e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:31.646703 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:31.646629    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:31.646859 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.646749    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:31.646859 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:31.646809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:31.646979 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.646887    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:31.646979 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:31.646943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:31.647095 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:31.647018    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:32.690027 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:32.689013    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" event={"ID":"693b0401-a9f9-41a4-b2e8-d3e3d6f37477","Type":"ContainerStarted","Data":"789ff9be36678b829ce7c675364e2835643879de056bbdb7e84446d6b786790e"}
Nov 01 12:21:32.700274 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:32.700240    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"839bae4638948a932829e431666671db","Type":"ContainerStarted","Data":"a3fa00c99728dfe064e0bca25339ac6d62572dc59a4676bc3f5fa2f1bb94fddf"}
Nov 01 12:21:33.647564 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:33.646854    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:33.647564 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:33.646987    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:33.647564 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:33.647354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:33.647564 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:33.647426    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:33.647564 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:33.647474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:33.647564 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:33.647526    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:33.704451 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:33.704419    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"d08d3cfd44df29731faafa69be6f8df6","Type":"ContainerStarted","Data":"c35c3ab8e4c7d4c441000795553d44e1b11b9eeee4d8dc323ef6dcb19e1d14f3"}
Nov 01 12:21:33.721622 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:33.721588    2582 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="789ff9be36678b829ce7c675364e2835643879de056bbdb7e84446d6b786790e" exitCode=0
Nov 01 12:21:33.721758 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:33.721696    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" event={"ID":"693b0401-a9f9-41a4-b2e8-d3e3d6f37477","Type":"ContainerDied","Data":"789ff9be36678b829ce7c675364e2835643879de056bbdb7e84446d6b786790e"}
Nov 01 12:21:33.731177 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:33.731147    2582 generic.go:356] "Generic (PLEG): container finished" podID="839bae4638948a932829e431666671db" containerID="a3fa00c99728dfe064e0bca25339ac6d62572dc59a4676bc3f5fa2f1bb94fddf" exitCode=0
Nov 01 12:21:33.731286 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:33.731198    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"839bae4638948a932829e431666671db","Type":"ContainerDied","Data":"a3fa00c99728dfe064e0bca25339ac6d62572dc59a4676bc3f5fa2f1bb94fddf"}
Nov 01 12:21:33.811337 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:33.811278    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal" podStartSLOduration=7.811257716 podStartE2EDuration="7.811257716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:33.810444431 +0000 UTC m=+7.761450814" watchObservedRunningTime="2026-11-01 12:21:33.811257716 +0000 UTC m=+7.762264099"
Nov 01 12:21:34.735496 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:34.735455    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bm5xm" event={"ID":"e5291295-5fd1-4e6e-b52b-c940c088168d","Type":"ContainerStarted","Data":"3a4103a0914172b2780ba24ef114199178627c212bfc18ad0b4f00d6827253d4"}
Nov 01 12:21:34.741894 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:34.741819    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"839bae4638948a932829e431666671db","Type":"ContainerStarted","Data":"1a925b09a5c452a9f252159e905fff6ce02448a09887f367581b4d1d39267406"}
Nov 01 12:21:34.955752 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:34.955646    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bm5xm" podStartSLOduration=4.168912375 podStartE2EDuration="8.955627222s" totalImagesPullingTime="4.786714847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:34.843666508 +0000 UTC m=+8.794672889" watchObservedRunningTime="2026-11-01 12:21:34.955627222 +0000 UTC m=+8.906633607"
Nov 01 12:21:35.342221 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.342181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:35.342372 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.342350    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:35.342436 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.342423    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret podName:370a3bfd-1661-4e44-b5d5-5f6b83db1f6c nodeName:}" failed. No retries permitted until 2026-11-01 12:21:43.342401249 +0000 UTC m=+17.293407625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret") pod "global-pull-secret-syncer-v7lhr" (UID: "370a3bfd-1661-4e44-b5d5-5f6b83db1f6c") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:35.444747 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.444186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:35.444747 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.444358    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:35.444747 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.444423    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs podName:3888b182-8fb4-49bd-9af3-873bcc3efa92 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:43.444401788 +0000 UTC m=+17.395408163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs") pod "network-metrics-daemon-p779s" (UID: "3888b182-8fb4-49bd-9af3-873bcc3efa92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:35.545206 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.544879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:35.545206 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.545094    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:35.545206 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.545117    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:35.545206 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.545130    2582 projected.go:196] Error preparing data for projected volume kube-api-access-x26dv for pod openshift-network-diagnostics/network-check-target-6zjj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:35.545206 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.545183    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv podName:47f53f35-b29e-4db5-91c8-0df05af362e7 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:43.545164934 +0000 UTC m=+17.496171299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x26dv" (UniqueName: "kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv") pod "network-check-target-6zjj4" (UID: "47f53f35-b29e-4db5-91c8-0df05af362e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:35.647269 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.646901    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:35.647269 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.646926    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:35.647269 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.646936    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:35.647269 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.647041    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:35.647269 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.647143    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:35.647269 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:35.647199    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:35.748353 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.748240    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5" event={"ID":"46433966-ea95-45b6-9092-c2f73bd7cd53","Type":"ContainerStarted","Data":"30b5c1874029fdf35e4dbd22ab369a540a75356054a2b9d79a91ca60a3cc0d5c"}
Nov 01 12:21:35.751279 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.751243    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fpq52" event={"ID":"f8b43ebe-6a94-45f6-a613-0931f4607e9e","Type":"ContainerStarted","Data":"cdfcee9c01c3c6e1672b6b00c5017157d2e5e56a7d7f0164582a34559af68de4"}
Nov 01 12:21:35.859075 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.859006    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal" podStartSLOduration=9.858987515 podStartE2EDuration="9.858987515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:34.956826994 +0000 UTC m=+8.907833371" watchObservedRunningTime="2026-11-01 12:21:35.858987515 +0000 UTC m=+9.809993896"
Nov 01 12:21:35.859365 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:35.859326    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fpq52" podStartSLOduration=3.9871432 podStartE2EDuration="9.859314863s" totalImagesPullingTime="5.872171663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:35.858422094 +0000 UTC m=+9.809428476" watchObservedRunningTime="2026-11-01 12:21:35.859314863 +0000 UTC m=+9.810321247"
Nov 01 12:21:36.982502 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:36.982468    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:36.983798 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:36.983474    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:37.647381 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.647351    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:37.647520 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:37.647470    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:37.647795 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.647766    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:37.647889 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:37.647869    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:37.647945 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.647929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:37.648030 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:37.648009    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:37.707300 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.707267    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 12:21:37.764023 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.763942    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kfdrq" event={"ID":"c09c0770-6cec-4827-9fb0-b07464653b27","Type":"ContainerStarted","Data":"99a797a8c95c6d79fd1edb003f9a0bc8a6913c40e4995242a67a24192c982c5f"}
Nov 01 12:21:37.766471 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.766441    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5" event={"ID":"46433966-ea95-45b6-9092-c2f73bd7cd53","Type":"ContainerStarted","Data":"b2d520be2d5100b8a7f9031d7b6f8778f39fbbdab89850b150efb9d384ed0ba7"}
Nov 01 12:21:37.768189 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.768163    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxjxw" event={"ID":"5cedf131-a296-4792-8328-f3ad030d3003","Type":"ContainerStarted","Data":"6c9a07c613a0af76041decf99cf244f3a79e3a108e0aa829713d1ab320add50a"}
Nov 01 12:21:37.768476 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.768457    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:37.768950 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.768906    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:21:37.980249 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.980188    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lxjxw" podStartSLOduration=3.629112206 podStartE2EDuration="11.98016875s" totalImagesPullingTime="8.351056544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:37.979652179 +0000 UTC m=+11.930658561" watchObservedRunningTime="2026-11-01 12:21:37.98016875 +0000 UTC m=+11.931175134"
Nov 01 12:21:37.980502 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:37.980471    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kfdrq" podStartSLOduration=3.6720223130000003 podStartE2EDuration="11.980463283s" totalImagesPullingTime="8.30844097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:37.872132395 +0000 UTC m=+11.823138778" watchObservedRunningTime="2026-11-01 12:21:37.980463283 +0000 UTC m=+11.931469666"
Nov 01 12:21:38.668273 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:38.668170    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T12:21:37.707288172Z","UUID":"0df2d3e0-0cfb-423f-9ddd-847debbf1819","Handler":null,"Name":"","Endpoint":""}
Nov 01 12:21:38.669741 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:38.669722    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 12:21:38.669741 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:38.669745    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 12:21:38.773768 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:38.773484    2582 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="ff4dc214db264f263966e2a7e917c20bcc86342ce1372f41c2b590ddef3499d9" exitCode=0
Nov 01 12:21:38.774160 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:38.773570    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" event={"ID":"693b0401-a9f9-41a4-b2e8-d3e3d6f37477","Type":"ContainerDied","Data":"ff4dc214db264f263966e2a7e917c20bcc86342ce1372f41c2b590ddef3499d9"}
Nov 01 12:21:39.647462 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:39.647385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:39.647612 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:39.647459    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:39.647612 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:39.647562    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:39.647612 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:39.647576    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:39.647737 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:39.647671    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:39.647796 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:39.647769    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:39.778572 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:39.778311    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xwv48" event={"ID":"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2","Type":"ContainerStarted","Data":"b375a1496edc553df39466839942cefcc3c08a2975e24d02b25574ae2f6224cc"}
Nov 01 12:21:39.780735 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:39.780705    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dvz6d" event={"ID":"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36","Type":"ContainerStarted","Data":"bcb5ffc94bc90a57cd3f809013b1345e61b9c8c53c4d7d78a90eb0b7617a3a6d"}
Nov 01 12:21:39.783369 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:39.783333    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5" event={"ID":"46433966-ea95-45b6-9092-c2f73bd7cd53","Type":"ContainerStarted","Data":"e1ab08342a28e76b604e723b20796b1606da9364af288fbb56e22963f11b55f1"}
Nov 01 12:21:39.885616 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:39.885566    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xwv48" podStartSLOduration=5.234823832 podStartE2EDuration="13.885553284s" totalImagesPullingTime="8.650729452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:39.885255328 +0000 UTC m=+13.836261713" watchObservedRunningTime="2026-11-01 12:21:39.885553284 +0000 UTC m=+13.836559665"
Nov 01 12:21:39.994505 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:39.994423    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5" podStartSLOduration=3.8794591670000003 podStartE2EDuration="13.994408581s" totalImagesPullingTime="10.114949414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:39.994224496 +0000 UTC m=+13.945230878" watchObservedRunningTime="2026-11-01 12:21:39.994408581 +0000 UTC m=+13.945414959"
Nov 01 12:21:40.102345 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:40.102289    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dvz6d" podStartSLOduration=3.713811771 podStartE2EDuration="14.102274872s" totalImagesPullingTime="10.388463101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:40.101993657 +0000 UTC m=+14.053000039" watchObservedRunningTime="2026-11-01 12:21:40.102274872 +0000 UTC m=+14.053281252"
Nov 01 12:21:40.788139 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:40.788058    2582 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="9c2941adbe17205cac228f982c46cf06d7b0cf951cfc2ea56d8bf4cea21f60b7" exitCode=0
Nov 01 12:21:40.788139 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:40.788078    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" event={"ID":"693b0401-a9f9-41a4-b2e8-d3e3d6f37477","Type":"ContainerDied","Data":"9c2941adbe17205cac228f982c46cf06d7b0cf951cfc2ea56d8bf4cea21f60b7"}
Nov 01 12:21:41.646956 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:41.646920    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:41.646956 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:41.646940    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:41.647222 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:41.647034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:41.647222 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:41.647035    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:41.647222 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:41.647149    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:41.647342 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:41.647237    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:42.795185 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:42.794933    2582 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="bdc2237f4c9d898370dd53fe23e40e9274b9605464b95f590bb71e5c48d0fa39" exitCode=0
Nov 01 12:21:42.795185 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:42.795031    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" event={"ID":"693b0401-a9f9-41a4-b2e8-d3e3d6f37477","Type":"ContainerDied","Data":"bdc2237f4c9d898370dd53fe23e40e9274b9605464b95f590bb71e5c48d0fa39"}
Nov 01 12:21:43.402524 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:43.402484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:43.402708 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.402645    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:43.402770 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.402722    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret podName:370a3bfd-1661-4e44-b5d5-5f6b83db1f6c nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.402698687 +0000 UTC m=+33.353705052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret") pod "global-pull-secret-syncer-v7lhr" (UID: "370a3bfd-1661-4e44-b5d5-5f6b83db1f6c") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:43.503337 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:43.503299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:43.503518 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.503479    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:43.503583 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.503552    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs podName:3888b182-8fb4-49bd-9af3-873bcc3efa92 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.503532257 +0000 UTC m=+33.454538633 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs") pod "network-metrics-daemon-p779s" (UID: "3888b182-8fb4-49bd-9af3-873bcc3efa92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:43.604831 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:43.604793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:43.605008 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.604983    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:43.605008 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.605008    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:43.605144 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.605020    2582 projected.go:196] Error preparing data for projected volume kube-api-access-x26dv for pod openshift-network-diagnostics/network-check-target-6zjj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:43.605144 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.605102    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv podName:47f53f35-b29e-4db5-91c8-0df05af362e7 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.605080823 +0000 UTC m=+33.556087197 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x26dv" (UniqueName: "kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv") pod "network-check-target-6zjj4" (UID: "47f53f35-b29e-4db5-91c8-0df05af362e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:43.646945 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:43.646914    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:43.647129 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:43.647024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:43.647129 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.647035    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:43.647129 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:43.647073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:43.647265 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.647158    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:43.647265 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:43.647253    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:45.647373 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.647203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:45.647816 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.647252    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:45.647816 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:45.647446    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:45.647816 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.647253    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:45.647816 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:45.647528    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:45.647816 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:45.647604    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:45.803863 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.803829    2582 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="ebfc6227506f940fd92df54b92a00ddd407817bd6864c4523ac15e90c9f86ebb" exitCode=0
Nov 01 12:21:45.804061 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.803904    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" event={"ID":"693b0401-a9f9-41a4-b2e8-d3e3d6f37477","Type":"ContainerDied","Data":"ebfc6227506f940fd92df54b92a00ddd407817bd6864c4523ac15e90c9f86ebb"}
Nov 01 12:21:45.807018 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.806999    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" event={"ID":"19e0e140-91e4-4d92-b1ac-0ea2115e3467","Type":"ContainerStarted","Data":"e21839574724aee0181def2f0ed7d7041e625757f4d94aaa0d7e55e54ec98d7f"}
Nov 01 12:21:45.807106 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.807074    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" event={"ID":"19e0e140-91e4-4d92-b1ac-0ea2115e3467","Type":"ContainerStarted","Data":"4c375317edfc86be00536b0f5b4ed14ee8ad8a3dbf4f1784f6477ffc994db11e"}
Nov 01 12:21:45.807150 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.807107    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" event={"ID":"19e0e140-91e4-4d92-b1ac-0ea2115e3467","Type":"ContainerStarted","Data":"aef70a1129fe8f4ddb0562998545b31e8cbb6e371dac532450ac9d13410e47d8"}
Nov 01 12:21:45.807150 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.807143    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" event={"ID":"19e0e140-91e4-4d92-b1ac-0ea2115e3467","Type":"ContainerStarted","Data":"1ec61c7ffadb61e4dd4010dbe4e05901bbe8e81a75e3b399b54c191c50b32f5a"}
Nov 01 12:21:45.807207 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.807170    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" event={"ID":"19e0e140-91e4-4d92-b1ac-0ea2115e3467","Type":"ContainerStarted","Data":"8304c183dc59e682653e44b07f710cdeb02a577ee0bae6e36417fc72a73ef51d"}
Nov 01 12:21:45.807207 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:45.807199    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" event={"ID":"19e0e140-91e4-4d92-b1ac-0ea2115e3467","Type":"ContainerStarted","Data":"37fb72e6a816518be1a9b807555e89f20dd28920679aa7ae484bb69c022f22e2"}
Nov 01 12:21:46.811577 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:46.811538    2582 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="1f6980da3ee79dc1ef05a975c87c4c3a1dfd5135a9e9c20d01ef8fd9af0b848b" exitCode=0
Nov 01 12:21:46.812148 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:46.811592    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" event={"ID":"693b0401-a9f9-41a4-b2e8-d3e3d6f37477","Type":"ContainerDied","Data":"1f6980da3ee79dc1ef05a975c87c4c3a1dfd5135a9e9c20d01ef8fd9af0b848b"}
Nov 01 12:21:47.646868 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:47.646779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:47.646868 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:47.646852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:47.647117 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:47.646878    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:47.647117 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:47.646965    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:47.647117 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:47.647095    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:47.647251 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:47.647162    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:47.817426 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:47.817395    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" event={"ID":"693b0401-a9f9-41a4-b2e8-d3e3d6f37477","Type":"ContainerStarted","Data":"b1e5c01c8f85f21c789ffd8211b9dd6ac9de8cadd4fed540abbbf0cc80fa1ce6"}
Nov 01 12:21:47.820335 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:47.820311    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" event={"ID":"19e0e140-91e4-4d92-b1ac-0ea2115e3467","Type":"ContainerStarted","Data":"c6a891f63abf4bac5eb28fcf010d1c276e3f0523f374981d9084db9f07ebbf5a"}
Nov 01 12:21:49.647288 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:49.647212    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:49.647848 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:49.647338    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:49.647848 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:49.647365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:49.647848 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:49.647364    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:49.647848 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:49.647435    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:49.647848 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:49.647511    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:49.828095 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:49.828040    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" event={"ID":"19e0e140-91e4-4d92-b1ac-0ea2115e3467","Type":"ContainerStarted","Data":"57b9f020029097a75a885baab63d80021f1acc3bf1cacc6877938fb6d2e9dcb9"}
Nov 01 12:21:49.828434 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:49.828407    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:49.828554 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:49.828468    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:49.843327 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:49.843300    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:50.084561 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:50.084320    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf" podStartSLOduration=8.022075841 podStartE2EDuration="24.084308085s" totalImagesPullingTime="16.062232244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:50.036906318 +0000 UTC m=+23.987912699" watchObservedRunningTime="2026-11-01 12:21:50.084308085 +0000 UTC m=+24.035314465"
Nov 01 12:21:50.084917 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:50.084894    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tg2vq" podStartSLOduration=10.863106283 podStartE2EDuration="24.084886756s" totalImagesPullingTime="13.221780473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:47.976736096 +0000 UTC m=+21.927742477" watchObservedRunningTime="2026-11-01 12:21:50.084886756 +0000 UTC m=+24.035893136"
Nov 01 12:21:50.830550 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:50.830522    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:50.845426 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:50.845405    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:21:51.647705 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:51.647675    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:51.647857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:51.647742    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:51.647857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:51.647838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:51.647923 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:51.647867    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:51.647954 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:51.647940    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:51.647993 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:51.647972    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:52.009317 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:52.009248    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7lhr"]
Nov 01 12:21:52.009670 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:52.009346    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:52.009670 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:52.009428    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:52.012136 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:52.012108    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p779s"]
Nov 01 12:21:52.012271 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:52.012239    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:52.012382 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:52.012346    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:52.012705 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:52.012681    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6zjj4"]
Nov 01 12:21:52.012768 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:52.012755    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:52.012835 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:52.012819    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:53.646612 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:53.646580    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:53.647108 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:53.646581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:53.647108 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:53.646705    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:53.647108 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:53.646582    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:53.647108 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:53.646770    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:53.647108 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:53.646829    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:55.647637 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:55.647550    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:55.648096 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:55.647574    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:55.648096 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:55.647667    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:55.648096 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:55.647679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:55.648096 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:55.647766    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:55.648096 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:55.647864    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:57.647366 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:57.647330    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:57.647835 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:57.647439    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:57.647835 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:57.647454    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6zjj4" podUID="47f53f35-b29e-4db5-91c8-0df05af362e7"
Nov 01 12:21:57.647835 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:57.647476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:57.647835 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:57.647586    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7lhr" podUID="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c"
Nov 01 12:21:57.647835 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:57.647689    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p779s" podUID="3888b182-8fb4-49bd-9af3-873bcc3efa92"
Nov 01 12:21:58.290471 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.290407    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeReady"
Nov 01 12:21:58.290601 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.290523    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 12:21:58.398921 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.398884    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rsltv"]
Nov 01 12:21:58.444519 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.444485    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tsmbq"]
Nov 01 12:21:58.444702 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.444597    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rsltv"
Nov 01 12:21:58.463101 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.463077    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fg5h6"]
Nov 01 12:21:58.463237 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.463131    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tsmbq"
Nov 01 12:21:58.479201 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.479179    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79dd7d46d9-xfhjz"]
Nov 01 12:21:58.479298 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.479289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fg5h6"
Nov 01 12:21:58.494518 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.494493    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.494518 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.494507    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.500095 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.500073    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rdmlg"]
Nov 01 12:21:58.500223 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.500154    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.509811 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.509786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6dv\" (UniqueName: \"kubernetes.io/projected/3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf-kube-api-access-6j6dv\") pod \"kas-connection-checker-685895c7c9-rsltv\" (UID: \"3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf\") " pod="kube-system/kas-connection-checker-685895c7c9-rsltv"
Nov 01 12:21:58.520262 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.520237    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"]
Nov 01 12:21:58.520380 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.520353    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.534494 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.534476    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84fbdf67f8-6fmbg"]
Nov 01 12:21:58.534632 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.534602    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:58.548775 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.548731    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"]
Nov 01 12:21:58.548893 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.548834    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.550404 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.550363    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 12:21:58.550652 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.550631    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 12:21:58.550733 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.550421    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 12:21:58.556064 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.551272    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-d6zhm\""
Nov 01 12:21:58.569103 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.569078    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-744lb"]
Nov 01 12:21:58.569212 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.569139    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.583214 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.583196    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"]
Nov 01 12:21:58.583319 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.583303    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:21:58.595039 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.595017    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 12:21:58.595244 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.595227    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.595293 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.595252    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 12:21:58.595345 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.595332    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.595399 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.595371    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.595470 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.595454    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.595526 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.595461    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 12:21:58.595578 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.595528    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.595578 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.595529    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.597993 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.597975    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 12:21:58.607363 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.607346    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"]
Nov 01 12:21:58.607468 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.607454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"
Nov 01 12:21:58.610171 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-service-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.610261 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s67j\" (UniqueName: \"kubernetes.io/projected/39facba4-434d-476e-9e84-a8388d78e4e9-kube-api-access-4s67j\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:58.610261 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610201    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhf84\" (UniqueName: \"kubernetes.io/projected/b6cb3539-d27f-4954-9b4d-e730dfd5809d-kube-api-access-qhf84\") pod \"kas-connection-checker-685895c7c9-fg5h6\" (UID: \"b6cb3539-d27f-4954-9b4d-e730dfd5809d\") " pod="kube-system/kas-connection-checker-685895c7c9-fg5h6"
Nov 01 12:21:58.610261 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:58.610372 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.610420 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-snapshots\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.610471 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-default-certificate\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.610520 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-tmp\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.610575 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:58.610617 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8lp6\" (UniqueName: \"kubernetes.io/projected/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-kube-api-access-m8lp6\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.610654 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6kk\" (UniqueName: \"kubernetes.io/projected/bad6344f-4c79-4140-abcc-2c4ae3d63049-kube-api-access-vc6kk\") pod \"kas-connection-checker-685895c7c9-tsmbq\" (UID: \"bad6344f-4c79-4140-abcc-2c4ae3d63049\") " pod="kube-system/kas-connection-checker-685895c7c9-tsmbq"
Nov 01 12:21:58.610692 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.610692 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-serving-cert\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.610764 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.610764 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6dv\" (UniqueName: \"kubernetes.io/projected/3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf-kube-api-access-6j6dv\") pod \"kas-connection-checker-685895c7c9-rsltv\" (UID: \"3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf\") " pod="kube-system/kas-connection-checker-685895c7c9-rsltv"
Nov 01 12:21:58.610849 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqv4d\" (UniqueName: \"kubernetes.io/projected/426e64a6-ecf0-471e-b130-15f85716445f-kube-api-access-pqv4d\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.610849 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.610833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-stats-auth\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.631434 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.631411    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"]
Nov 01 12:21:58.631621 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.631602    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.643119 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.643096    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 12:21:58.643226 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.643147    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 12:21:58.643226 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.643196    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.643226 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.643211    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.643332 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.643233    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 12:21:58.645817 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.645803    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-569b8d479f-htb2l"]
Nov 01 12:21:58.645930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.645915    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:58.650929 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.650886    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 12:21:58.651327 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651042    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 12:21:58.651327 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651102    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 12:21:58.651327 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651145    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 12:21:58.651591 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651380    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-m68cz\""
Nov 01 12:21:58.651591 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651521    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kqn6l\""
Nov 01 12:21:58.651591 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651524    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 12:21:58.651724 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651656    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 12:21:58.651724 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651712    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8q978\""
Nov 01 12:21:58.651807 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651788    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wkt5z\""
Nov 01 12:21:58.651938 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651922    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 12:21:58.651983 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.651975    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jtrgb\""
Nov 01 12:21:58.656702 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.656687    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.656702 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.656693    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.656817 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.656694    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-v2t2t\""
Nov 01 12:21:58.665268 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.665249    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6dv\" (UniqueName: \"kubernetes.io/projected/3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf-kube-api-access-6j6dv\") pod \"kas-connection-checker-685895c7c9-rsltv\" (UID: \"3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf\") " pod="kube-system/kas-connection-checker-685895c7c9-rsltv"
Nov 01 12:21:58.666550 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.666531    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-slc7h"]
Nov 01 12:21:58.666650 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.666637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.680825 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.680807    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 12:21:58.680904 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.680827    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.681004 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.680984    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.681122 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.681037    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.681122 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.681045    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 12:21:58.681394 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.681380    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jxlnp\""
Nov 01 12:21:58.684266 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.684248    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"]
Nov 01 12:21:58.695837 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.695804    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.695837 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.695831    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.696039 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.696023    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 12:21:58.696891 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.696875    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xm57l"]
Nov 01 12:21:58.697006 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.696993    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"
Nov 01 12:21:58.711312 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:58.711441 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.711441 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvmj\" (UniqueName: \"kubernetes.io/projected/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-kube-api-access-kzvmj\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:58.711441 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.711441 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c00596c9-ffa3-461b-81be-7315388deaf1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:21:58.711596 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.711460    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 12:21:58.711596 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-tmp\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.711596 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.711532    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.211512884 +0000 UTC m=+33.162519264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : secret "router-metrics-certs-default" not found
Nov 01 12:21:58.711596 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rjm\" (UniqueName: \"kubernetes.io/projected/690056c7-6829-4caf-b59e-e2650d20bb47-kube-api-access-54rjm\") pod \"volume-data-source-validator-58c9f5f46d-j4vzd\" (UID: \"690056c7-6829-4caf-b59e-e2650d20bb47\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"
Nov 01 12:21:58.711789 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.711789 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.711789 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:21:58.711789 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:58.711789 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8lp6\" (UniqueName: \"kubernetes.io/projected/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-kube-api-access-m8lp6\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.711789 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711779    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-tmp\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.712109 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6kk\" (UniqueName: \"kubernetes.io/projected/bad6344f-4c79-4140-abcc-2c4ae3d63049-kube-api-access-vc6kk\") pod \"kas-connection-checker-685895c7c9-tsmbq\" (UID: \"bad6344f-4c79-4140-abcc-2c4ae3d63049\") " pod="kube-system/kas-connection-checker-685895c7c9-tsmbq"
Nov 01 12:21:58.712109 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.712109 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711901    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-config\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.712109 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-stats-auth\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.712109 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.711992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:58.712109 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.712109 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-service-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.712109 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.712427 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712110    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:58.712427 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhf84\" (UniqueName: \"kubernetes.io/projected/b6cb3539-d27f-4954-9b4d-e730dfd5809d-kube-api-access-qhf84\") pod \"kas-connection-checker-685895c7c9-fg5h6\" (UID: \"b6cb3539-d27f-4954-9b4d-e730dfd5809d\") " pod="kube-system/kas-connection-checker-685895c7c9-fg5h6"
Nov 01 12:21:58.712427 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rjb\" (UniqueName: \"kubernetes.io/projected/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-kube-api-access-x2rjb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.712427 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-serving-cert\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.712427 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-snapshots\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.712427 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-default-certificate\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.712610 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:58.712610 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.712610 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-serving-cert\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.712610 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.712610 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqv4d\" (UniqueName: \"kubernetes.io/projected/426e64a6-ecf0-471e-b130-15f85716445f-kube-api-access-pqv4d\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.712849 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.712849 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712622    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-service-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.712849 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.712849 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wr4q\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.712849 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.712703    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 12:21:58.712849 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s67j\" (UniqueName: \"kubernetes.io/projected/39facba4-434d-476e-9e84-a8388d78e4e9-kube-api-access-4s67j\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:58.712849 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.712849 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7csw\" (UniqueName: \"kubernetes.io/projected/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-kube-api-access-h7csw\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.712849 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.712776    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls podName:39facba4-434d-476e-9e84-a8388d78e4e9 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.21276076 +0000 UTC m=+33.163767133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lq5xn" (UID: "39facba4-434d-476e-9e84-a8388d78e4e9") : secret "samples-operator-tls" not found
Nov 01 12:21:58.713309 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712903    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-snapshots\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.713309 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.712973    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.713309 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.713006    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.212992414 +0000 UTC m=+33.163998795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : configmap references non-existent config key: service-ca.crt
Nov 01 12:21:58.714568 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.714548    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-stats-auth\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.714734 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.714716    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-default-certificate\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.714799 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.714753    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-serving-cert\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.721278 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.721259    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x6gnr"]
Nov 01 12:21:58.721396 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.721380    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.750988 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.750963    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 12:21:58.751144 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751090    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-62ffm\""
Nov 01 12:21:58.751144 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751112    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 12:21:58.751144 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751131    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-c2rwl\""
Nov 01 12:21:58.751144 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751136    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fj2dt\""
Nov 01 12:21:58.751489 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751475    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rsltv"]
Nov 01 12:21:58.751584 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751501    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tsmbq"]
Nov 01 12:21:58.751584 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751516    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fg5h6"]
Nov 01 12:21:58.751584 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751534    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79dd7d46d9-xfhjz"]
Nov 01 12:21:58.751584 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751549    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"]
Nov 01 12:21:58.751584 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751566    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"]
Nov 01 12:21:58.751584 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751579    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rdmlg"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751592    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751604    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-slc7h"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751618    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-569b8d479f-htb2l"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751700    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751726    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-744lb"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751746    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751772    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84fbdf67f8-6fmbg"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751798    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751816    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x6gnr"]
Nov 01 12:21:58.751857 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.751835    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xm57l"]
Nov 01 12:21:58.766831 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.766802    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6kk\" (UniqueName: \"kubernetes.io/projected/bad6344f-4c79-4140-abcc-2c4ae3d63049-kube-api-access-vc6kk\") pod \"kas-connection-checker-685895c7c9-tsmbq\" (UID: \"bad6344f-4c79-4140-abcc-2c4ae3d63049\") " pod="kube-system/kas-connection-checker-685895c7c9-tsmbq"
Nov 01 12:21:58.767089 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.767066    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhf84\" (UniqueName: \"kubernetes.io/projected/b6cb3539-d27f-4954-9b4d-e730dfd5809d-kube-api-access-qhf84\") pod \"kas-connection-checker-685895c7c9-fg5h6\" (UID: \"b6cb3539-d27f-4954-9b4d-e730dfd5809d\") " pod="kube-system/kas-connection-checker-685895c7c9-fg5h6"
Nov 01 12:21:58.767362 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.767341    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s67j\" (UniqueName: \"kubernetes.io/projected/39facba4-434d-476e-9e84-a8388d78e4e9-kube-api-access-4s67j\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:58.767403 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.767360    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqv4d\" (UniqueName: \"kubernetes.io/projected/426e64a6-ecf0-471e-b130-15f85716445f-kube-api-access-pqv4d\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:58.767515 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.767501    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8lp6\" (UniqueName: \"kubernetes.io/projected/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-kube-api-access-m8lp6\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.773373 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.773333    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rsltv"
Nov 01 12:21:58.784482 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.784463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tsmbq"
Nov 01 12:21:58.795975 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.795958    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.796066 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.796005    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.796111 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.796097    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.796111 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.796105    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 12:21:58.796171 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.796125    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.796276 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.796219    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 12:21:58.796878 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.796864    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 12:21:58.800332 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.800284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fg5h6"
Nov 01 12:21:58.800486 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.800435    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 12:21:58.800536 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.800522    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 12:21:58.813361 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.813453 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7csw\" (UniqueName: \"kubernetes.io/projected/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-kube-api-access-h7csw\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.813453 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvmj\" (UniqueName: \"kubernetes.io/projected/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-kube-api-access-kzvmj\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:58.813453 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.813453 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c00596c9-ffa3-461b-81be-7315388deaf1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:21:58.813623 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rjm\" (UniqueName: \"kubernetes.io/projected/690056c7-6829-4caf-b59e-e2650d20bb47-kube-api-access-54rjm\") pod \"volume-data-source-validator-58c9f5f46d-j4vzd\" (UID: \"690056c7-6829-4caf-b59e-e2650d20bb47\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"
Nov 01 12:21:58.813623 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.813623 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.813623 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjsj\" (UniqueName: \"kubernetes.io/projected/9b41f9fc-afce-464d-a18c-330694f80140-kube-api-access-6rjsj\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.813623 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-tmp-dir\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.813841 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8pw\" (UniqueName: \"kubernetes.io/projected/3953b512-91d0-401c-8397-0b9314e7a87b-kube-api-access-rk8pw\") pod \"network-check-source-6f78d59f99-nwjwn\" (UID: \"3953b512-91d0-401c-8397-0b9314e7a87b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"
Nov 01 12:21:58.813841 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:21:58.813841 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:58.813970 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.813865    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 12:21:58.813970 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.813924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7da0156a-c68b-4e26-8344-852632f8c78e-ca-trust-extracted\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.814091 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.813977    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls podName:3758a7d1-ccc2-4322-bd01-e742e5c11ff4 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.313932041 +0000 UTC m=+33.264938420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jbpml" (UID: "3758a7d1-ccc2-4322-bd01-e742e5c11ff4") : secret "cluster-monitoring-operator-tls" not found
Nov 01 12:21:58.814091 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.813996    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 12:21:58.814091 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814024    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c00596c9-ffa3-461b-81be-7315388deaf1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:21:58.814091 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.814080    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert podName:c00596c9-ffa3-461b-81be-7315388deaf1 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.314043559 +0000 UTC m=+33.265049937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-744lb" (UID: "c00596c9-ffa3-461b-81be-7315388deaf1") : secret "networking-console-plugin-cert" not found
Nov 01 12:21:58.814282 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-config\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.814282 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814158    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f772\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-kube-api-access-6f772\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.814282 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814189    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.814282 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:58.814282 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.814510 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.814510 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-trusted-ca\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.814510 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-installation-pull-secrets\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.814510 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rjb\" (UniqueName: \"kubernetes.io/projected/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-kube-api-access-x2rjb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.814510 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-serving-cert\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.814510 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-registry-certificates\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-trusted-ca\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814563    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-config\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814603    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-config-volume\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mxw6\" (UniqueName: \"kubernetes.io/projected/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-kube-api-access-4mxw6\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-bound-sa-token\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814727    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-config\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b41f9fc-afce-464d-a18c-330694f80140-serving-cert\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814756    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-image-registry-private-configuration\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.814787 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.815381 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.815381 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wr4q\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.815381 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.815381 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.814928    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:21:58.815381 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.814939    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fbdf67f8-6fmbg: secret "image-registry-tls" not found
Nov 01 12:21:58.815381 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.814990    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls podName:97f50f8c-9c08-406f-aba6-83fc9a2f75fd nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.314973954 +0000 UTC m=+33.265980313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls") pod "image-registry-84fbdf67f8-6fmbg" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd") : secret "image-registry-tls" not found
Nov 01 12:21:58.815381 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814994    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:58.815684 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.815535    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.815684 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.814593    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.816242 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.816224    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.816939 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.816915    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-serving-cert\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.817208 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.817183    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.817289 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.817276    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.841547 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.841515    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:21:58.854272 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.854238    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 12:21:58.854403 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.854328    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 12:21:58.854403 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.854359    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lt5kb\""
Nov 01 12:21:58.854403 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.854369    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-86gmg\""
Nov 01 12:21:58.868307 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.868279    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7csw\" (UniqueName: \"kubernetes.io/projected/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-kube-api-access-h7csw\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.868610 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.868587    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvmj\" (UniqueName: \"kubernetes.io/projected/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-kube-api-access-kzvmj\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:58.869738 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.869718    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.870245 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.870226    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wr4q\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:58.870483 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.870467    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rjm\" (UniqueName: \"kubernetes.io/projected/690056c7-6829-4caf-b59e-e2650d20bb47-kube-api-access-54rjm\") pod \"volume-data-source-validator-58c9f5f46d-j4vzd\" (UID: \"690056c7-6829-4caf-b59e-e2650d20bb47\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"
Nov 01 12:21:58.870652 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.870581    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rjb\" (UniqueName: \"kubernetes.io/projected/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-kube-api-access-x2rjb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.916110 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.916250 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkxh\" (UniqueName: \"kubernetes.io/projected/34c9bd1f-837b-4c0c-83da-3e5b09856707-kube-api-access-6bkxh\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:21:58.916250 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjsj\" (UniqueName: \"kubernetes.io/projected/9b41f9fc-afce-464d-a18c-330694f80140-kube-api-access-6rjsj\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.916250 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-tmp-dir\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.916250 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8pw\" (UniqueName: \"kubernetes.io/projected/3953b512-91d0-401c-8397-0b9314e7a87b-kube-api-access-rk8pw\") pod \"network-check-source-6f78d59f99-nwjwn\" (UID: \"3953b512-91d0-401c-8397-0b9314e7a87b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"
Nov 01 12:21:58.916250 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7da0156a-c68b-4e26-8344-852632f8c78e-ca-trust-extracted\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.916250 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34c9bd1f-837b-4c0c-83da-3e5b09856707-cert\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f772\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-kube-api-access-6f772\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-trusted-ca\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-installation-pull-secrets\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-registry-certificates\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-trusted-ca\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-config\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-config-volume\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxw6\" (UniqueName: \"kubernetes.io/projected/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-kube-api-access-4mxw6\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-bound-sa-token\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.916457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b41f9fc-afce-464d-a18c-330694f80140-serving-cert\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.916772 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.916464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-image-registry-private-configuration\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.917022 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.917000    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-tmp-dir\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.917163 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.917118    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 12:21:58.917216 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.917179    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls podName:ee4ddc2b-35a6-4862-b64c-bb137efcdd69 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.417159542 +0000 UTC m=+33.368165915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls") pod "dns-default-xm57l" (UID: "ee4ddc2b-35a6-4862-b64c-bb137efcdd69") : secret "dns-default-metrics-tls" not found
Nov 01 12:21:58.917364 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.917344    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-trusted-ca\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.917842 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.917820    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-config\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.917932 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.917888    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7da0156a-c68b-4e26-8344-852632f8c78e-ca-trust-extracted\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.918231 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.918213    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-registry-certificates\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.918850 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.918831    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-trusted-ca\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.918930 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.918895    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-image-registry-private-configuration\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.918930 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.918921    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:21:58.919027 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.918935    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569b8d479f-htb2l: secret "image-registry-tls" not found
Nov 01 12:21:58.919027 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:58.918978    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls podName:7da0156a-c68b-4e26-8344-852632f8c78e nodeName:}" failed. No retries permitted until 2026-11-01 12:21:59.418962861 +0000 UTC m=+33.369969247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls") pod "image-registry-569b8d479f-htb2l" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e") : secret "image-registry-tls" not found
Nov 01 12:21:58.919208 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.919033    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-config-volume\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.919571 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.919554    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-installation-pull-secrets\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.920827 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.920809    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b41f9fc-afce-464d-a18c-330694f80140-serving-cert\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.943161 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.943126    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:21:58.955686 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.955661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:21:58.977019 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.976992    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-bound-sa-token\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:58.977239 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.977212    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8pw\" (UniqueName: \"kubernetes.io/projected/3953b512-91d0-401c-8397-0b9314e7a87b-kube-api-access-rk8pw\") pod \"network-check-source-6f78d59f99-nwjwn\" (UID: \"3953b512-91d0-401c-8397-0b9314e7a87b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"
Nov 01 12:21:58.977362 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.977345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"
Nov 01 12:21:58.977782 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.977684    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjsj\" (UniqueName: \"kubernetes.io/projected/9b41f9fc-afce-464d-a18c-330694f80140-kube-api-access-6rjsj\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:58.978411 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.978388    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxw6\" (UniqueName: \"kubernetes.io/projected/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-kube-api-access-4mxw6\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:58.978619 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:58.978600    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f772\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-kube-api-access-6f772\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:59.020654 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.017660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34c9bd1f-837b-4c0c-83da-3e5b09856707-cert\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:21:59.020654 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.019239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkxh\" (UniqueName: \"kubernetes.io/projected/34c9bd1f-837b-4c0c-83da-3e5b09856707-kube-api-access-6bkxh\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:21:59.024122 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.024074    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34c9bd1f-837b-4c0c-83da-3e5b09856707-cert\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:21:59.044355 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.044329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"
Nov 01 12:21:59.062437 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.062407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:21:59.077457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.076737    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkxh\" (UniqueName: \"kubernetes.io/projected/34c9bd1f-837b-4c0c-83da-3e5b09856707-kube-api-access-6bkxh\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:21:59.092859 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.092823    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:21:59.215731 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.215699    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tsmbq"]
Nov 01 12:21:59.218312 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.218288    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rsltv"]
Nov 01 12:21:59.221864 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.221763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:59.221864 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.221852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:21:59.221968 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.221928    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:00.221910981 +0000 UTC m=+34.172917373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : configmap references non-existent config key: service-ca.crt
Nov 01 12:21:59.222026 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.221995    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 12:21:59.222088 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.222063    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:00.222029487 +0000 UTC m=+34.173035876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : secret "router-metrics-certs-default" not found
Nov 01 12:21:59.222088 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.222029    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:21:59.222187 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.222119    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 12:21:59.222187 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.222152    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls podName:39facba4-434d-476e-9e84-a8388d78e4e9 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:00.222141784 +0000 UTC m=+34.173148146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lq5xn" (UID: "39facba4-434d-476e-9e84-a8388d78e4e9") : secret "samples-operator-tls" not found
Nov 01 12:21:59.224960 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:59.224937    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad6344f_4c79_4140_abcc_2c4ae3d63049.slice/crio-bbe766f5bec5c46aaa06e6ea37b0424e2f12dd74f5865c0d225e4e6df89050d8 WatchSource:0}: Error finding container bbe766f5bec5c46aaa06e6ea37b0424e2f12dd74f5865c0d225e4e6df89050d8: Status 404 returned error can't find the container with id bbe766f5bec5c46aaa06e6ea37b0424e2f12dd74f5865c0d225e4e6df89050d8
Nov 01 12:21:59.226870 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:59.226848    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8c18aa_2a61_4ddb_b0cf_75c0b76e45bf.slice/crio-f3d41b277973477bd26c72f0c47fbb5091275143533952735ede44b700b6ee37 WatchSource:0}: Error finding container f3d41b277973477bd26c72f0c47fbb5091275143533952735ede44b700b6ee37: Status 404 returned error can't find the container with id f3d41b277973477bd26c72f0c47fbb5091275143533952735ede44b700b6ee37
Nov 01 12:21:59.280865 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.280818    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rdmlg"]
Nov 01 12:21:59.294348 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:59.294310    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1efc2ea_a886_4c13_97a7_b1ddbd621feb.slice/crio-5f6cfad4d2a416533c267b61ca1f75a9dbbf3e8efd36ad921a71a3591c5177cf WatchSource:0}: Error finding container 5f6cfad4d2a416533c267b61ca1f75a9dbbf3e8efd36ad921a71a3591c5177cf: Status 404 returned error can't find the container with id 5f6cfad4d2a416533c267b61ca1f75a9dbbf3e8efd36ad921a71a3591c5177cf
Nov 01 12:21:59.323131 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.323103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:21:59.323255 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.323205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:21:59.323255 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.323237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:21:59.323521 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.323465    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 12:21:59.323521 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.323490    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 12:21:59.323645 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.323501    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:21:59.323645 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.323571    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fbdf67f8-6fmbg: secret "image-registry-tls" not found
Nov 01 12:21:59.323645 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.323545    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls podName:3758a7d1-ccc2-4322-bd01-e742e5c11ff4 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:00.323515329 +0000 UTC m=+34.274521710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jbpml" (UID: "3758a7d1-ccc2-4322-bd01-e742e5c11ff4") : secret "cluster-monitoring-operator-tls" not found
Nov 01 12:21:59.323645 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.323638    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert podName:c00596c9-ffa3-461b-81be-7315388deaf1 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:00.323624346 +0000 UTC m=+34.274630711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-744lb" (UID: "c00596c9-ffa3-461b-81be-7315388deaf1") : secret "networking-console-plugin-cert" not found
Nov 01 12:21:59.323869 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.323660    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls podName:97f50f8c-9c08-406f-aba6-83fc9a2f75fd nodeName:}" failed. No retries permitted until 2026-11-01 12:22:00.32365053 +0000 UTC m=+34.274656890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls") pod "image-registry-84fbdf67f8-6fmbg" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd") : secret "image-registry-tls" not found
Nov 01 12:21:59.333017 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.332994    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"]
Nov 01 12:21:59.343332 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.343307    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"]
Nov 01 12:21:59.354816 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:59.354788    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2ec5b4_ed5e_49fa_ac4a_dbca006ed843.slice/crio-4cef24fe79149cdd7d5c18280de7fe567075cd1a6ecec869b74f34b5f17fe665 WatchSource:0}: Error finding container 4cef24fe79149cdd7d5c18280de7fe567075cd1a6ecec869b74f34b5f17fe665: Status 404 returned error can't find the container with id 4cef24fe79149cdd7d5c18280de7fe567075cd1a6ecec869b74f34b5f17fe665
Nov 01 12:21:59.377938 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.377832    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fg5h6"]
Nov 01 12:21:59.386183 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:59.386158    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cb3539_d27f_4954_9b4d_e730dfd5809d.slice/crio-7b1b9c0c9cc7dd8e7d04dd704ffc64bffc986a97bac42c4a6a8835321eb59e18 WatchSource:0}: Error finding container 7b1b9c0c9cc7dd8e7d04dd704ffc64bffc986a97bac42c4a6a8835321eb59e18: Status 404 returned error can't find the container with id 7b1b9c0c9cc7dd8e7d04dd704ffc64bffc986a97bac42c4a6a8835321eb59e18
Nov 01 12:21:59.387675 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.387654    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"]
Nov 01 12:21:59.396619 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:59.396592    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690056c7_6829_4caf_b59e_e2650d20bb47.slice/crio-62e8634a9944589861c55c9744227e50c8ca411f4566a85ef8cd3802fb43bc0f WatchSource:0}: Error finding container 62e8634a9944589861c55c9744227e50c8ca411f4566a85ef8cd3802fb43bc0f: Status 404 returned error can't find the container with id 62e8634a9944589861c55c9744227e50c8ca411f4566a85ef8cd3802fb43bc0f
Nov 01 12:21:59.417745 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.417721    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"]
Nov 01 12:21:59.424129 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.424103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:21:59.424377 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.424344    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 12:21:59.424377 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.424357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:59.424534 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.424411    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls podName:ee4ddc2b-35a6-4862-b64c-bb137efcdd69 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:00.424389202 +0000 UTC m=+34.375395570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls") pod "dns-default-xm57l" (UID: "ee4ddc2b-35a6-4862-b64c-bb137efcdd69") : secret "dns-default-metrics-tls" not found
Nov 01 12:21:59.424534 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.424451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:21:59.424534 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.424480    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:59.424688 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.424540    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret podName:370a3bfd-1661-4e44-b5d5-5f6b83db1f6c nodeName:}" failed. No retries permitted until 2026-11-01 12:22:31.424523078 +0000 UTC m=+65.375529453 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret") pod "global-pull-secret-syncer-v7lhr" (UID: "370a3bfd-1661-4e44-b5d5-5f6b83db1f6c") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:59.424688 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.424568    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:21:59.424688 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.424581    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569b8d479f-htb2l: secret "image-registry-tls" not found
Nov 01 12:21:59.424688 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.424622    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls podName:7da0156a-c68b-4e26-8344-852632f8c78e nodeName:}" failed. No retries permitted until 2026-11-01 12:22:00.424611117 +0000 UTC m=+34.375617497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls") pod "image-registry-569b8d479f-htb2l" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e") : secret "image-registry-tls" not found
Nov 01 12:21:59.434171 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:59.434144    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3953b512_91d0_401c_8397_0b9314e7a87b.slice/crio-28e2f6de099644659469607c349c0bd5570004e29c98a262e6f239abbf7e0a44 WatchSource:0}: Error finding container 28e2f6de099644659469607c349c0bd5570004e29c98a262e6f239abbf7e0a44: Status 404 returned error can't find the container with id 28e2f6de099644659469607c349c0bd5570004e29c98a262e6f239abbf7e0a44
Nov 01 12:21:59.454641 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.454615    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-slc7h"]
Nov 01 12:21:59.463031 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:59.463003    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b41f9fc_afce_464d_a18c_330694f80140.slice/crio-e90a2419d0c439d98c6bbe650e374cc26457eaf6b79fb89f8d1bf90dc041032f WatchSource:0}: Error finding container e90a2419d0c439d98c6bbe650e374cc26457eaf6b79fb89f8d1bf90dc041032f: Status 404 returned error can't find the container with id e90a2419d0c439d98c6bbe650e374cc26457eaf6b79fb89f8d1bf90dc041032f
Nov 01 12:21:59.488611 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.488585    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x6gnr"]
Nov 01 12:21:59.496858 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:21:59.496835    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c9bd1f_837b_4c0c_83da_3e5b09856707.slice/crio-533dfbadc7dd1c10cf42b78f0108ef1a2fb34aa2d1c1af95ef5fd5340cf3dbb5 WatchSource:0}: Error finding container 533dfbadc7dd1c10cf42b78f0108ef1a2fb34aa2d1c1af95ef5fd5340cf3dbb5: Status 404 returned error can't find the container with id 533dfbadc7dd1c10cf42b78f0108ef1a2fb34aa2d1c1af95ef5fd5340cf3dbb5
Nov 01 12:21:59.525451 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.525418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:59.525569 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.525555    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:59.525614 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:21:59.525605    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs podName:3888b182-8fb4-49bd-9af3-873bcc3efa92 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:31.525592482 +0000 UTC m=+65.476598844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs") pod "network-metrics-daemon-p779s" (UID: "3888b182-8fb4-49bd-9af3-873bcc3efa92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:59.626656 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.626571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:59.629088 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.629045    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:59.646888 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.646860    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:59.646988 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.646951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:21:59.646988 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.646977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:21:59.696461 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.696434    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 12:21:59.699292 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.699274    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 12:21:59.796843 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.796808    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9t2fc\""
Nov 01 12:21:59.796843 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.796808    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzkjs\""
Nov 01 12:21:59.797033 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.796878    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-phch9\""
Nov 01 12:21:59.800979 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.800965    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:21:59.851299 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.851257    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x6gnr" event={"ID":"34c9bd1f-837b-4c0c-83da-3e5b09856707","Type":"ContainerStarted","Data":"533dfbadc7dd1c10cf42b78f0108ef1a2fb34aa2d1c1af95ef5fd5340cf3dbb5"}
Nov 01 12:21:59.852587 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.852556    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" event={"ID":"9b41f9fc-afce-464d-a18c-330694f80140","Type":"ContainerStarted","Data":"e90a2419d0c439d98c6bbe650e374cc26457eaf6b79fb89f8d1bf90dc041032f"}
Nov 01 12:21:59.853845 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.853820    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd" event={"ID":"690056c7-6829-4caf-b59e-e2650d20bb47","Type":"ContainerStarted","Data":"62e8634a9944589861c55c9744227e50c8ca411f4566a85ef8cd3802fb43bc0f"}
Nov 01 12:21:59.855493 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.855466    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p" event={"ID":"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843","Type":"ContainerStarted","Data":"4cef24fe79149cdd7d5c18280de7fe567075cd1a6ecec869b74f34b5f17fe665"}
Nov 01 12:21:59.856711 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.856685    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh" event={"ID":"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c","Type":"ContainerStarted","Data":"8100acd04afd92612cb431b9ebf075b8d2d6d5de959dd4276468bf44fe4d78b0"}
Nov 01 12:21:59.857823 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.857799    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rdmlg" event={"ID":"d1efc2ea-a886-4c13-97a7-b1ddbd621feb","Type":"ContainerStarted","Data":"5f6cfad4d2a416533c267b61ca1f75a9dbbf3e8efd36ad921a71a3591c5177cf"}
Nov 01 12:21:59.859317 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.859295    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fg5h6" event={"ID":"b6cb3539-d27f-4954-9b4d-e730dfd5809d","Type":"ContainerStarted","Data":"0a9e5ce6232e0af265d1975e6db1f732a304a65ed2c26b5f0fc5b464657c0c2b"}
Nov 01 12:21:59.859410 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.859331    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fg5h6" event={"ID":"b6cb3539-d27f-4954-9b4d-e730dfd5809d","Type":"ContainerStarted","Data":"7b1b9c0c9cc7dd8e7d04dd704ffc64bffc986a97bac42c4a6a8835321eb59e18"}
Nov 01 12:21:59.860735 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.860713    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tsmbq" event={"ID":"bad6344f-4c79-4140-abcc-2c4ae3d63049","Type":"ContainerStarted","Data":"38e57925b57c9234c31afd3dee3d3c7aa8e2ed10a2b4eeed41d9ba5bd2d1f73a"}
Nov 01 12:21:59.860836 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.860750    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tsmbq" event={"ID":"bad6344f-4c79-4140-abcc-2c4ae3d63049","Type":"ContainerStarted","Data":"bbe766f5bec5c46aaa06e6ea37b0424e2f12dd74f5865c0d225e4e6df89050d8"}
Nov 01 12:21:59.861856 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.861832    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn" event={"ID":"3953b512-91d0-401c-8397-0b9314e7a87b","Type":"ContainerStarted","Data":"28e2f6de099644659469607c349c0bd5570004e29c98a262e6f239abbf7e0a44"}
Nov 01 12:21:59.863241 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.863219    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rsltv" event={"ID":"3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf","Type":"ContainerStarted","Data":"1b97f9daa6af01f2fe8f772a616e13c20f81f0a1dd52fe583461ba7c58d1805e"}
Nov 01 12:21:59.863319 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:21:59.863255    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rsltv" event={"ID":"3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf","Type":"ContainerStarted","Data":"f3d41b277973477bd26c72f0c47fbb5091275143533952735ede44b700b6ee37"}
Nov 01 12:22:00.186555 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.186515    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6zjj4"]
Nov 01 12:22:00.195109 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:22:00.195082    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f53f35_b29e_4db5_91c8_0df05af362e7.slice/crio-524fba8833ffac4a77f7472fa799f01603d6041c70eae7eaf6278c512c9c9dd1 WatchSource:0}: Error finding container 524fba8833ffac4a77f7472fa799f01603d6041c70eae7eaf6278c512c9c9dd1: Status 404 returned error can't find the container with id 524fba8833ffac4a77f7472fa799f01603d6041c70eae7eaf6278c512c9c9dd1
Nov 01 12:22:00.231368 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.231341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:00.231521 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.231389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:00.231521 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.231457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:00.231521 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.231510    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 12:22:00.231659 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.231579    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls podName:39facba4-434d-476e-9e84-a8388d78e4e9 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:02.231555645 +0000 UTC m=+36.182562020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lq5xn" (UID: "39facba4-434d-476e-9e84-a8388d78e4e9") : secret "samples-operator-tls" not found
Nov 01 12:22:00.231659 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.231596    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:02.231589499 +0000 UTC m=+36.182595857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : configmap references non-existent config key: service-ca.crt
Nov 01 12:22:00.231777 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.231659    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 12:22:00.231777 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.231723    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:02.231704828 +0000 UTC m=+36.182711204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : secret "router-metrics-certs-default" not found
Nov 01 12:22:00.332380 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.332351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:00.332530 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.332411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:00.332530 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.332431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:00.332530 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.332496    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:22:00.332530 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.332517    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fbdf67f8-6fmbg: secret "image-registry-tls" not found
Nov 01 12:22:00.332530 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.332525    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 12:22:00.332694 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.332548    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 12:22:00.332694 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.332573    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls podName:3758a7d1-ccc2-4322-bd01-e742e5c11ff4 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:02.332560858 +0000 UTC m=+36.283567230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jbpml" (UID: "3758a7d1-ccc2-4322-bd01-e742e5c11ff4") : secret "cluster-monitoring-operator-tls" not found
Nov 01 12:22:00.332694 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.332585    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls podName:97f50f8c-9c08-406f-aba6-83fc9a2f75fd nodeName:}" failed. No retries permitted until 2026-11-01 12:22:02.332579624 +0000 UTC m=+36.283585983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls") pod "image-registry-84fbdf67f8-6fmbg" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd") : secret "image-registry-tls" not found
Nov 01 12:22:00.332694 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.332596    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert podName:c00596c9-ffa3-461b-81be-7315388deaf1 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:02.332589829 +0000 UTC m=+36.283596187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-744lb" (UID: "c00596c9-ffa3-461b-81be-7315388deaf1") : secret "networking-console-plugin-cert" not found
Nov 01 12:22:00.424384 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.424328    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rsltv" podStartSLOduration=443.424314282 podStartE2EDuration="7m23.424314282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:00.423228282 +0000 UTC m=+34.374234664" watchObservedRunningTime="2026-11-01 12:22:00.424314282 +0000 UTC m=+34.375320663"
Nov 01 12:22:00.433633 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.433594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:00.433799 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.433704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:22:00.433799 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.433782    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 12:22:00.433905 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.433812    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:22:00.433905 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.433823    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569b8d479f-htb2l: secret "image-registry-tls" not found
Nov 01 12:22:00.433905 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.433851    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls podName:ee4ddc2b-35a6-4862-b64c-bb137efcdd69 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:02.433832398 +0000 UTC m=+36.384838774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls") pod "dns-default-xm57l" (UID: "ee4ddc2b-35a6-4862-b64c-bb137efcdd69") : secret "dns-default-metrics-tls" not found
Nov 01 12:22:00.433905 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:00.433870    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls podName:7da0156a-c68b-4e26-8344-852632f8c78e nodeName:}" failed. No retries permitted until 2026-11-01 12:22:02.433861175 +0000 UTC m=+36.384867539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls") pod "image-registry-569b8d479f-htb2l" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e") : secret "image-registry-tls" not found
Nov 01 12:22:00.640644 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.640514    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tsmbq" podStartSLOduration=443.640490009 podStartE2EDuration="7m23.640490009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:00.531449863 +0000 UTC m=+34.482456242" watchObservedRunningTime="2026-11-01 12:22:00.640490009 +0000 UTC m=+34.591496389"
Nov 01 12:22:00.875039 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:00.874940    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6zjj4" event={"ID":"47f53f35-b29e-4db5-91c8-0df05af362e7","Type":"ContainerStarted","Data":"524fba8833ffac4a77f7472fa799f01603d6041c70eae7eaf6278c512c9c9dd1"}
Nov 01 12:22:02.253543 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:02.253502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:02.254154 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:02.253727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:02.254154 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:02.253778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:02.254154 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.253928    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.253909885 +0000 UTC m=+40.204916258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : configmap references non-existent config key: service-ca.crt
Nov 01 12:22:02.254444 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.254402    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 12:22:02.254503 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.254454    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.254438289 +0000 UTC m=+40.205444670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : secret "router-metrics-certs-default" not found
Nov 01 12:22:02.254597 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.254581    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 12:22:02.254660 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.254629    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls podName:39facba4-434d-476e-9e84-a8388d78e4e9 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.254617829 +0000 UTC m=+40.205624210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lq5xn" (UID: "39facba4-434d-476e-9e84-a8388d78e4e9") : secret "samples-operator-tls" not found
Nov 01 12:22:02.354826 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:02.354771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:02.354826 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:02.354833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:02.355078 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.354937    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 12:22:02.355078 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:02.354965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:02.355078 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.355018    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert podName:c00596c9-ffa3-461b-81be-7315388deaf1 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.354997965 +0000 UTC m=+40.306004344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-744lb" (UID: "c00596c9-ffa3-461b-81be-7315388deaf1") : secret "networking-console-plugin-cert" not found
Nov 01 12:22:02.355253 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.355170    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 12:22:02.355253 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.355193    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:22:02.355253 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.355210    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fbdf67f8-6fmbg: secret "image-registry-tls" not found
Nov 01 12:22:02.355253 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.355212    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls podName:3758a7d1-ccc2-4322-bd01-e742e5c11ff4 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.35519974 +0000 UTC m=+40.306206160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jbpml" (UID: "3758a7d1-ccc2-4322-bd01-e742e5c11ff4") : secret "cluster-monitoring-operator-tls" not found
Nov 01 12:22:02.355442 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.355275    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls podName:97f50f8c-9c08-406f-aba6-83fc9a2f75fd nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.355259733 +0000 UTC m=+40.306266094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls") pod "image-registry-84fbdf67f8-6fmbg" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd") : secret "image-registry-tls" not found
Nov 01 12:22:02.455955 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:02.455919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:22:02.456156 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:02.455992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:02.456156 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.456131    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:22:02.456272 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.456150    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 12:22:02.456272 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.456252    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls podName:ee4ddc2b-35a6-4862-b64c-bb137efcdd69 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.456231292 +0000 UTC m=+40.407237672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls") pod "dns-default-xm57l" (UID: "ee4ddc2b-35a6-4862-b64c-bb137efcdd69") : secret "dns-default-metrics-tls" not found
Nov 01 12:22:02.456272 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.456158    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569b8d479f-htb2l: secret "image-registry-tls" not found
Nov 01 12:22:02.456419 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:02.456332    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls podName:7da0156a-c68b-4e26-8344-852632f8c78e nodeName:}" failed. No retries permitted until 2026-11-01 12:22:06.456309318 +0000 UTC m=+40.407315716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls") pod "image-registry-569b8d479f-htb2l" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e") : secret "image-registry-tls" not found
Nov 01 12:22:03.895042 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:03.894958    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd" event={"ID":"690056c7-6829-4caf-b59e-e2650d20bb47","Type":"ContainerStarted","Data":"bef5ad114eeae3822d595b3175fc8424ce7122c06959e844933dba771adc14ee"}
Nov 01 12:22:04.003036 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:04.002982    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fg5h6" podStartSLOduration=447.002967845 podStartE2EDuration="7m27.002967845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:00.640210681 +0000 UTC m=+34.591217067" watchObservedRunningTime="2026-11-01 12:22:04.002967845 +0000 UTC m=+37.953974225"
Nov 01 12:22:05.903356 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:05.903315    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p" event={"ID":"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843","Type":"ContainerStarted","Data":"744941c4a9eca47c8f7844eb7dfe4453fa33d1c5657dd2163566adeb0dedf89c"}
Nov 01 12:22:05.905911 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:05.905882    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh" event={"ID":"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c","Type":"ContainerStarted","Data":"7e6625e5252700cbf041b994b922fe0636ff81bf4c7b78574534af120f8e8dc6"}
Nov 01 12:22:05.907600 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:05.907562    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rdmlg" event={"ID":"d1efc2ea-a886-4c13-97a7-b1ddbd621feb","Type":"ContainerStarted","Data":"a54f6554c65034f35162f1d6145dc39afe8de0006ae274b827a05812983f2333"}
Nov 01 12:22:05.909517 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:05.909495    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/0.log"
Nov 01 12:22:05.909634 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:05.909536    2582 generic.go:356] "Generic (PLEG): container finished" podID="9b41f9fc-afce-464d-a18c-330694f80140" containerID="2404857b63d5e7240d81aeee8fa2297fd19ff776988594841d9b9169027e4f7f" exitCode=255
Nov 01 12:22:05.909634 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:05.909567    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" event={"ID":"9b41f9fc-afce-464d-a18c-330694f80140","Type":"ContainerDied","Data":"2404857b63d5e7240d81aeee8fa2297fd19ff776988594841d9b9169027e4f7f"}
Nov 01 12:22:05.909809 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:05.909791    2582 scope.go:122] "RemoveContainer" containerID="2404857b63d5e7240d81aeee8fa2297fd19ff776988594841d9b9169027e4f7f"
Nov 01 12:22:06.011430 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.011376    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p" podStartSLOduration=396.113783685 podStartE2EDuration="6m42.011359024s" totalImagesPullingTime="5.897575339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:06.010517062 +0000 UTC m=+39.961523445" watchObservedRunningTime="2026-11-01 12:22:06.011359024 +0000 UTC m=+39.962365407"
Nov 01 12:22:06.012859 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.012820    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd" podStartSLOduration=399.589208535 podStartE2EDuration="6m43.012806537s" totalImagesPullingTime="3.423598002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:04.00387032 +0000 UTC m=+37.954876701" watchObservedRunningTime="2026-11-01 12:22:06.012806537 +0000 UTC m=+39.963812918"
Nov 01 12:22:06.230656 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.230566    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh" podStartSLOduration=398.336445579 podStartE2EDuration="6m44.230551829s" totalImagesPullingTime="5.89410625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:06.120652207 +0000 UTC m=+40.071658591" watchObservedRunningTime="2026-11-01 12:22:06.230551829 +0000 UTC m=+40.181558210"
Nov 01 12:22:06.231587 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.231556    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-rdmlg" podStartSLOduration=401.59804136 podStartE2EDuration="6m47.23154402s" totalImagesPullingTime="5.63350266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:06.229882768 +0000 UTC m=+40.180889151" watchObservedRunningTime="2026-11-01 12:22:06.23154402 +0000 UTC m=+40.182550402"
Nov 01 12:22:06.304083 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.303435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:06.304083 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.303583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:06.304083 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.303627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:06.304083 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.303803    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.303785158 +0000 UTC m=+48.254791520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : configmap references non-existent config key: service-ca.crt
Nov 01 12:22:06.304083 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.303895    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 12:22:06.304083 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.303935    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.303923601 +0000 UTC m=+48.254929962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : secret "router-metrics-certs-default" not found
Nov 01 12:22:06.304083 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.304001    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 12:22:06.304083 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.304037    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls podName:39facba4-434d-476e-9e84-a8388d78e4e9 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.304025055 +0000 UTC m=+48.255031416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lq5xn" (UID: "39facba4-434d-476e-9e84-a8388d78e4e9") : secret "samples-operator-tls" not found
Nov 01 12:22:06.326202 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.326173    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9xgxt"]
Nov 01 12:22:06.350498 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.350477    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.403994 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.403967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03ef7e88-664f-413b-9424-c16a88d3b58c-data-volume\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.404141 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.404014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrg8\" (UniqueName: \"kubernetes.io/projected/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-api-access-hhrg8\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.404141 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.404098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:06.404141 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.404130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.404291 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.404166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.404291 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.404172    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:22:06.404291 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.404190    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fbdf67f8-6fmbg: secret "image-registry-tls" not found
Nov 01 12:22:06.404291 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.404204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:06.404291 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.404221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:06.404291 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.404233    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls podName:97f50f8c-9c08-406f-aba6-83fc9a2f75fd nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.404217837 +0000 UTC m=+48.355224199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls") pod "image-registry-84fbdf67f8-6fmbg" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd") : secret "image-registry-tls" not found
Nov 01 12:22:06.404555 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.404337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03ef7e88-664f-413b-9424-c16a88d3b58c-crio-socket\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.404555 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.404362    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 12:22:06.404555 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.404368    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 12:22:06.404555 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.404388    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls podName:3758a7d1-ccc2-4322-bd01-e742e5c11ff4 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.404381157 +0000 UTC m=+48.355387516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jbpml" (UID: "3758a7d1-ccc2-4322-bd01-e742e5c11ff4") : secret "cluster-monitoring-operator-tls" not found
Nov 01 12:22:06.404555 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.404428    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert podName:c00596c9-ffa3-461b-81be-7315388deaf1 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.404409083 +0000 UTC m=+48.355415442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-744lb" (UID: "c00596c9-ffa3-461b-81be-7315388deaf1") : secret "networking-console-plugin-cert" not found
Nov 01 12:22:06.412430 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.412405    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9xgxt"]
Nov 01 12:22:06.471637 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.471612    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 12:22:06.471637 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.471618    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 12:22:06.471842 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.471649    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-thbc5\""
Nov 01 12:22:06.505001 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.504931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrg8\" (UniqueName: \"kubernetes.io/projected/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-api-access-hhrg8\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.505001 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.504976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:06.505001 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.504996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.505270 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.505084    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 12:22:06.505270 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.505118    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 12:22:06.505270 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.505130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.505270 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.505144    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm podName:03ef7e88-664f-413b-9424-c16a88d3b58c nodeName:}" failed. No retries permitted until 2026-11-01 12:22:07.005127435 +0000 UTC m=+40.956133813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xgxt" (UID: "03ef7e88-664f-413b-9424-c16a88d3b58c") : configmap "kube-rbac-proxy" not found
Nov 01 12:22:06.505270 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.505171    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls podName:ee4ddc2b-35a6-4862-b64c-bb137efcdd69 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.505155812 +0000 UTC m=+48.456162185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls") pod "dns-default-xm57l" (UID: "ee4ddc2b-35a6-4862-b64c-bb137efcdd69") : secret "dns-default-metrics-tls" not found
Nov 01 12:22:06.505270 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.505196    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:06.505270 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.505265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03ef7e88-664f-413b-9424-c16a88d3b58c-crio-socket\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.505541 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.505296    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls podName:03ef7e88-664f-413b-9424-c16a88d3b58c nodeName:}" failed. No retries permitted until 2026-11-01 12:22:07.005287537 +0000 UTC m=+40.956293897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xgxt" (UID: "03ef7e88-664f-413b-9424-c16a88d3b58c") : secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:06.505541 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.505338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:22:06.505541 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.505338    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03ef7e88-664f-413b-9424-c16a88d3b58c-crio-socket\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.505541 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.505380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03ef7e88-664f-413b-9424-c16a88d3b58c-data-volume\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.505541 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.505394    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:22:06.505541 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.505407    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569b8d479f-htb2l: secret "image-registry-tls" not found
Nov 01 12:22:06.505541 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.505442    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls podName:7da0156a-c68b-4e26-8344-852632f8c78e nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.505428458 +0000 UTC m=+48.456434832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls") pod "image-registry-569b8d479f-htb2l" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e") : secret "image-registry-tls" not found
Nov 01 12:22:06.505754 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.505625    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03ef7e88-664f-413b-9424-c16a88d3b58c-data-volume\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.558935 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.558909    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrg8\" (UniqueName: \"kubernetes.io/projected/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-api-access-hhrg8\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:06.919799 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.919696    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn" event={"ID":"3953b512-91d0-401c-8397-0b9314e7a87b","Type":"ContainerStarted","Data":"d859d7a93a22f908dd888ec8e2d096837ee9a842c7ae21d918156b043b9adc13"}
Nov 01 12:22:06.921628 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.921376    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6zjj4" event={"ID":"47f53f35-b29e-4db5-91c8-0df05af362e7","Type":"ContainerStarted","Data":"aa5ff44a4c0a48a08fe3fa7351b12bdb95b6df9219ccb110950952375bdbfe99"}
Nov 01 12:22:06.921628 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.921434    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:22:06.923024 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.922994    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x6gnr" event={"ID":"34c9bd1f-837b-4c0c-83da-3e5b09856707","Type":"ContainerStarted","Data":"cebe82a9e6a3088fb954bb5d3e566413b6d774a469c8385ff57917f67caba5ce"}
Nov 01 12:22:06.924559 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.924539    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/1.log"
Nov 01 12:22:06.924963 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.924948    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/0.log"
Nov 01 12:22:06.925079 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.924979    2582 generic.go:356] "Generic (PLEG): container finished" podID="9b41f9fc-afce-464d-a18c-330694f80140" containerID="0423994ee83ae3fc14143a43975b90f90d5886df399403a965d1b32428db2665" exitCode=255
Nov 01 12:22:06.925079 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.925008    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" event={"ID":"9b41f9fc-afce-464d-a18c-330694f80140","Type":"ContainerDied","Data":"0423994ee83ae3fc14143a43975b90f90d5886df399403a965d1b32428db2665"}
Nov 01 12:22:06.925199 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.925078    2582 scope.go:122] "RemoveContainer" containerID="2404857b63d5e7240d81aeee8fa2297fd19ff776988594841d9b9169027e4f7f"
Nov 01 12:22:06.925302 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:06.925286    2582 scope.go:122] "RemoveContainer" containerID="0423994ee83ae3fc14143a43975b90f90d5886df399403a965d1b32428db2665"
Nov 01 12:22:06.925492 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:06.925466    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-slc7h_openshift-console-operator(9b41f9fc-afce-464d-a18c-330694f80140)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" podUID="9b41f9fc-afce-464d-a18c-330694f80140"
Nov 01 12:22:07.010518 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.009833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:07.010518 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.009915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:07.010518 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:07.010154    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:07.010518 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:07.010216    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls podName:03ef7e88-664f-413b-9424-c16a88d3b58c nodeName:}" failed. No retries permitted until 2026-11-01 12:22:08.010197356 +0000 UTC m=+41.961203733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xgxt" (UID: "03ef7e88-664f-413b-9424-c16a88d3b58c") : secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:07.010518 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:07.010401    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 12:22:07.010518 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:07.010453    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm podName:03ef7e88-664f-413b-9424-c16a88d3b58c nodeName:}" failed. No retries permitted until 2026-11-01 12:22:08.010438955 +0000 UTC m=+41.961445315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xgxt" (UID: "03ef7e88-664f-413b-9424-c16a88d3b58c") : configmap "kube-rbac-proxy" not found
Nov 01 12:22:07.062492 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.062443    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn" podStartSLOduration=395.215992671 podStartE2EDuration="6m42.062430482s" totalImagesPullingTime="6.846437811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:07.061699356 +0000 UTC m=+41.012705743" watchObservedRunningTime="2026-11-01 12:22:07.062430482 +0000 UTC m=+41.013436863"
Nov 01 12:22:07.273885 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.273814    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"]
Nov 01 12:22:07.282114 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.282044    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x6gnr" podStartSLOduration=2.826230001 podStartE2EDuration="9.282030956s" totalImagesPullingTime="6.455800955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:07.281281052 +0000 UTC m=+41.232287433" watchObservedRunningTime="2026-11-01 12:22:07.282030956 +0000 UTC m=+41.233037337"
Nov 01 12:22:07.301711 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.301679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"
Nov 01 12:22:07.312853 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.312829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjtg\" (UniqueName: \"kubernetes.io/projected/4cf38c23-496f-4661-85ce-586416959d95-kube-api-access-jsjtg\") pod \"migrator-66df5d496b-twpxk\" (UID: \"4cf38c23-496f-4661-85ce-586416959d95\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"
Nov 01 12:22:07.331083 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.330187    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"]
Nov 01 12:22:07.387799 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.387729    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6zjj4" podStartSLOduration=34.909575631 podStartE2EDuration="41.387710489s" totalImagesPullingTime="6.478134858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:07.387635221 +0000 UTC m=+41.338641602" watchObservedRunningTime="2026-11-01 12:22:07.387710489 +0000 UTC m=+41.338716871"
Nov 01 12:22:07.413569 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.413539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjtg\" (UniqueName: \"kubernetes.io/projected/4cf38c23-496f-4661-85ce-586416959d95-kube-api-access-jsjtg\") pod \"migrator-66df5d496b-twpxk\" (UID: \"4cf38c23-496f-4661-85ce-586416959d95\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"
Nov 01 12:22:07.413819 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.413798    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-svx22\""
Nov 01 12:22:07.413914 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.413805    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:07.413914 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.413868    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 12:22:07.467673 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.467644    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjtg\" (UniqueName: \"kubernetes.io/projected/4cf38c23-496f-4661-85ce-586416959d95-kube-api-access-jsjtg\") pod \"migrator-66df5d496b-twpxk\" (UID: \"4cf38c23-496f-4661-85ce-586416959d95\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"
Nov 01 12:22:07.625884 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.625805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"
Nov 01 12:22:07.930092 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.930010    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/1.log"
Nov 01 12:22:07.930537 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:07.930340    2582 scope.go:122] "RemoveContainer" containerID="0423994ee83ae3fc14143a43975b90f90d5886df399403a965d1b32428db2665"
Nov 01 12:22:07.930611 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:07.930553    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-slc7h_openshift-console-operator(9b41f9fc-afce-464d-a18c-330694f80140)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" podUID="9b41f9fc-afce-464d-a18c-330694f80140"
Nov 01 12:22:08.011620 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:08.011589    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"]
Nov 01 12:22:08.018755 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:08.018726    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:08.018875 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:08.018785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:08.018939 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:08.018906    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 12:22:08.018993 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:08.018972    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm podName:03ef7e88-664f-413b-9424-c16a88d3b58c nodeName:}" failed. No retries permitted until 2026-11-01 12:22:10.018953674 +0000 UTC m=+43.969960035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9xgxt" (UID: "03ef7e88-664f-413b-9424-c16a88d3b58c") : configmap "kube-rbac-proxy" not found
Nov 01 12:22:08.019068 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:08.019006    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:08.019140 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:08.019102    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls podName:03ef7e88-664f-413b-9424-c16a88d3b58c nodeName:}" failed. No retries permitted until 2026-11-01 12:22:10.01908477 +0000 UTC m=+43.970091148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xgxt" (UID: "03ef7e88-664f-413b-9424-c16a88d3b58c") : secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:08.020411 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:22:08.020386    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf38c23_496f_4661_85ce_586416959d95.slice/crio-b8dd251ce1d9c2ca22b64a503d4545c365aa77e777504adfae1d628ce90e6b64 WatchSource:0}: Error finding container b8dd251ce1d9c2ca22b64a503d4545c365aa77e777504adfae1d628ce90e6b64: Status 404 returned error can't find the container with id b8dd251ce1d9c2ca22b64a503d4545c365aa77e777504adfae1d628ce90e6b64
Nov 01 12:22:08.934418 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:08.934387    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk" event={"ID":"4cf38c23-496f-4661-85ce-586416959d95","Type":"ContainerStarted","Data":"b8dd251ce1d9c2ca22b64a503d4545c365aa77e777504adfae1d628ce90e6b64"}
Nov 01 12:22:09.063323 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:09.063288    2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:09.063476 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:09.063343    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:09.063655 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:09.063643    2582 scope.go:122] "RemoveContainer" containerID="0423994ee83ae3fc14143a43975b90f90d5886df399403a965d1b32428db2665"
Nov 01 12:22:09.063829 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:09.063812    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-slc7h_openshift-console-operator(9b41f9fc-afce-464d-a18c-330694f80140)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" podUID="9b41f9fc-afce-464d-a18c-330694f80140"
Nov 01 12:22:10.039080 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:10.039030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:10.039452 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:10.039094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:10.039452 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:10.039264    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:10.039452 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:10.039331    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls podName:03ef7e88-664f-413b-9424-c16a88d3b58c nodeName:}" failed. No retries permitted until 2026-11-01 12:22:14.039313833 +0000 UTC m=+47.990320208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xgxt" (UID: "03ef7e88-664f-413b-9424-c16a88d3b58c") : secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:10.039549 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:10.039530    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:10.944514 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:10.944478    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk" event={"ID":"4cf38c23-496f-4661-85ce-586416959d95","Type":"ContainerStarted","Data":"8da32ac4bd8f241781884d09542f04f7a79b6298d571fc555854c6aa51d9b644"}
Nov 01 12:22:11.052752 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:11.052698    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk" podStartSLOduration=1.810168159 podStartE2EDuration="4.052685162s" totalImagesPullingTime="2.242517003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:11.051468322 +0000 UTC m=+45.002474702" watchObservedRunningTime="2026-11-01 12:22:11.052685162 +0000 UTC m=+45.003691542"
Nov 01 12:22:13.531591 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:13.531564    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kfdrq_c09c0770-6cec-4827-9fb0-b07464653b27/dns-node-resolver/0.log"
Nov 01 12:22:14.075819 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.075780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:14.075997 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.075927    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:14.075997 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.075993    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls podName:03ef7e88-664f-413b-9424-c16a88d3b58c nodeName:}" failed. No retries permitted until 2026-11-01 12:22:22.075977827 +0000 UTC m=+56.026984202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9xgxt" (UID: "03ef7e88-664f-413b-9424-c16a88d3b58c") : secret "insights-runtime-extractor-tls" not found
Nov 01 12:22:14.332550 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.332481    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fpq52_f8b43ebe-6a94-45f6-a613-0931f4607e9e/node-ca/0.log"
Nov 01 12:22:14.378165 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.378123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:14.378276 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.378182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:14.378276 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.378229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:14.378347 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.378275    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 12:22:14.378347 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.378318    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 12:22:14.378347 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.378332    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls podName:39facba4-434d-476e-9e84-a8388d78e4e9 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:30.378318257 +0000 UTC m=+64.329324643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lq5xn" (UID: "39facba4-434d-476e-9e84-a8388d78e4e9") : secret "samples-operator-tls" not found
Nov 01 12:22:14.378347 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.378344    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:30.378338991 +0000 UTC m=+64.329345350 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : configmap references non-existent config key: service-ca.crt
Nov 01 12:22:14.378475 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.378429    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs podName:426e64a6-ecf0-471e-b130-15f85716445f nodeName:}" failed. No retries permitted until 2026-11-01 12:22:30.378413756 +0000 UTC m=+64.329420127 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs") pod "router-default-79dd7d46d9-xfhjz" (UID: "426e64a6-ecf0-471e-b130-15f85716445f") : secret "router-metrics-certs-default" not found
Nov 01 12:22:14.479103 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.479073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:14.479232 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.479151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:14.479232 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.479201    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:22:14.479232 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.479220    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84fbdf67f8-6fmbg: secret "image-registry-tls" not found
Nov 01 12:22:14.479327 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.479238    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 12:22:14.479327 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.479250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:14.479327 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.479277    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 12:22:14.479327 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.479279    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls podName:97f50f8c-9c08-406f-aba6-83fc9a2f75fd nodeName:}" failed. No retries permitted until 2026-11-01 12:22:30.47926174 +0000 UTC m=+64.430268102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls") pod "image-registry-84fbdf67f8-6fmbg" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd") : secret "image-registry-tls" not found
Nov 01 12:22:14.479435 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.479338    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls podName:3758a7d1-ccc2-4322-bd01-e742e5c11ff4 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:30.479324144 +0000 UTC m=+64.430330503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jbpml" (UID: "3758a7d1-ccc2-4322-bd01-e742e5c11ff4") : secret "cluster-monitoring-operator-tls" not found
Nov 01 12:22:14.479435 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.479356    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert podName:c00596c9-ffa3-461b-81be-7315388deaf1 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:30.479350376 +0000 UTC m=+64.430356735 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-744lb" (UID: "c00596c9-ffa3-461b-81be-7315388deaf1") : secret "networking-console-plugin-cert" not found
Nov 01 12:22:14.580028 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.579997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:22:14.580414 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:14.580097    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:14.580414 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.580137    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 12:22:14.580414 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.580155    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569b8d479f-htb2l: secret "image-registry-tls" not found
Nov 01 12:22:14.580414 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.580202    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls podName:7da0156a-c68b-4e26-8344-852632f8c78e nodeName:}" failed. No retries permitted until 2026-11-01 12:22:30.580186814 +0000 UTC m=+64.531193173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls") pod "image-registry-569b8d479f-htb2l" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e") : secret "image-registry-tls" not found
Nov 01 12:22:14.580414 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.580221    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 12:22:14.580414 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:14.580263    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls podName:ee4ddc2b-35a6-4862-b64c-bb137efcdd69 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:30.580248732 +0000 UTC m=+64.531255105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls") pod "dns-default-xm57l" (UID: "ee4ddc2b-35a6-4862-b64c-bb137efcdd69") : secret "dns-default-metrics-tls" not found
Nov 01 12:22:15.333579 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:15.333548    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x6gnr_34c9bd1f-837b-4c0c-83da-3e5b09856707/serve-healthcheck-canary/0.log"
Nov 01 12:22:15.732613 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:15.732581    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-twpxk_4cf38c23-496f-4661-85ce-586416959d95/migrator/0.log"
Nov 01 12:22:15.937022 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:15.936988    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-xg58p_3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843/kube-storage-version-migrator-operator/0.log"
Nov 01 12:22:18.647789 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:18.647743    2582 scope.go:122] "RemoveContainer" containerID="0423994ee83ae3fc14143a43975b90f90d5886df399403a965d1b32428db2665"
Nov 01 12:22:18.971069 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:18.971035    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:22:18.971484 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:18.971467    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/1.log"
Nov 01 12:22:18.971557 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:18.971500    2582 generic.go:356] "Generic (PLEG): container finished" podID="9b41f9fc-afce-464d-a18c-330694f80140" containerID="df2d582fa3b0ee2aa06cd65a280dac02c8b043b6d7a355aef887ebb44c671f0c" exitCode=255
Nov 01 12:22:18.971592 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:18.971575    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" event={"ID":"9b41f9fc-afce-464d-a18c-330694f80140","Type":"ContainerDied","Data":"df2d582fa3b0ee2aa06cd65a280dac02c8b043b6d7a355aef887ebb44c671f0c"}
Nov 01 12:22:18.971639 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:18.971628    2582 scope.go:122] "RemoveContainer" containerID="0423994ee83ae3fc14143a43975b90f90d5886df399403a965d1b32428db2665"
Nov 01 12:22:18.971958 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:18.971942    2582 scope.go:122] "RemoveContainer" containerID="df2d582fa3b0ee2aa06cd65a280dac02c8b043b6d7a355aef887ebb44c671f0c"
Nov 01 12:22:18.972173 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:18.972152    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-slc7h_openshift-console-operator(9b41f9fc-afce-464d-a18c-330694f80140)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" podUID="9b41f9fc-afce-464d-a18c-330694f80140"
Nov 01 12:22:19.063650 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:19.063605    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:19.063821 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:19.063676    2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:19.977258 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:19.977230    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:22:19.977628 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:19.977545    2582 scope.go:122] "RemoveContainer" containerID="df2d582fa3b0ee2aa06cd65a280dac02c8b043b6d7a355aef887ebb44c671f0c"
Nov 01 12:22:19.977748 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:19.977729    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-slc7h_openshift-console-operator(9b41f9fc-afce-464d-a18c-330694f80140)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" podUID="9b41f9fc-afce-464d-a18c-330694f80140"
Nov 01 12:22:22.148805 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:22.148765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:22.151232 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:22.151209    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:22.325743 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:22.325702    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:22.694680 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:22.694645    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9xgxt"]
Nov 01 12:22:22.735947 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:22:22.735917    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ef7e88_664f_413b_9424_c16a88d3b58c.slice/crio-2a18ed8bb12322dbc7c4ee48db29d33cf28a4e5956e5f20f4908ddea6af352e3 WatchSource:0}: Error finding container 2a18ed8bb12322dbc7c4ee48db29d33cf28a4e5956e5f20f4908ddea6af352e3: Status 404 returned error can't find the container with id 2a18ed8bb12322dbc7c4ee48db29d33cf28a4e5956e5f20f4908ddea6af352e3
Nov 01 12:22:22.845351 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:22.845325    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:22.987553 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:22.987521    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xgxt" event={"ID":"03ef7e88-664f-413b-9424-c16a88d3b58c","Type":"ContainerStarted","Data":"e475550b91e9c99e329c242a56653e781bd7583b8c1b15df4ad493c27a10596f"}
Nov 01 12:22:22.987713 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:22.987578    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xgxt" event={"ID":"03ef7e88-664f-413b-9424-c16a88d3b58c","Type":"ContainerStarted","Data":"2a18ed8bb12322dbc7c4ee48db29d33cf28a4e5956e5f20f4908ddea6af352e3"}
Nov 01 12:22:24.994689 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:24.994654    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xgxt" event={"ID":"03ef7e88-664f-413b-9424-c16a88d3b58c","Type":"ContainerStarted","Data":"0bd22576241fc46f2742fafdecdd745a51ae0195f00ac417249db0456e107a3f"}
Nov 01 12:22:27.003378 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:27.003340    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9xgxt" event={"ID":"03ef7e88-664f-413b-9424-c16a88d3b58c","Type":"ContainerStarted","Data":"d77a64a9c775cbd9b8114e18677680365f1ede7e5f1be6c532aa717fdf34f61b"}
Nov 01 12:22:27.112251 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:27.112200    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9xgxt" podStartSLOduration=17.780904934 podStartE2EDuration="21.112186905s" totalImagesPullingTime="3.331281971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:22:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:27.111655143 +0000 UTC m=+61.062661537" watchObservedRunningTime="2026-11-01 12:22:27.112186905 +0000 UTC m=+61.063193280"
Nov 01 12:22:29.103631 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:29.103583    2582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-569b8d479f-htb2l"]
Nov 01 12:22:29.104163 ip-10-0-138-76 kubenswrapper[2582]: E1101 12:22:29.104078    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-569b8d479f-htb2l" podUID="7da0156a-c68b-4e26-8344-852632f8c78e"
Nov 01 12:22:30.013399 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.013370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:22:30.020660 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.020634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:22:30.125461 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.125428    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7da0156a-c68b-4e26-8344-852632f8c78e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7da0156a-c68b-4e26-8344-852632f8c78e-ca-trust-extracted\") pod \"7da0156a-c68b-4e26-8344-852632f8c78e\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") "
Nov 01 12:22:30.125897 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.125484    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-kube-api-access-6f772\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-kube-api-access-6f772\") pod \"7da0156a-c68b-4e26-8344-852632f8c78e\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") "
Nov 01 12:22:30.125897 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.125506    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-image-registry-private-configuration\") pod \"7da0156a-c68b-4e26-8344-852632f8c78e\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") "
Nov 01 12:22:30.125897 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.125558    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-bound-sa-token\") pod \"7da0156a-c68b-4e26-8344-852632f8c78e\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") "
Nov 01 12:22:30.125897 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.125574    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-installation-pull-secrets\") pod \"7da0156a-c68b-4e26-8344-852632f8c78e\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") "
Nov 01 12:22:30.125897 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.125594    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-registry-certificates\") pod \"7da0156a-c68b-4e26-8344-852632f8c78e\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") "
Nov 01 12:22:30.125897 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.125638    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-trusted-ca\") pod \"7da0156a-c68b-4e26-8344-852632f8c78e\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") "
Nov 01 12:22:30.125897 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.125756    2582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7da0156a-c68b-4e26-8344-852632f8c78e-ca-trust-extracted" pod "7da0156a-c68b-4e26-8344-852632f8c78e" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:22:30.126497 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.126002    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7da0156a-c68b-4e26-8344-852632f8c78e-ca-trust-extracted\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:22:30.126497 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.126154    2582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-trusted-ca" pod "7da0156a-c68b-4e26-8344-852632f8c78e" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:22:30.126497 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.126175    2582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-registry-certificates" pod "7da0156a-c68b-4e26-8344-852632f8c78e" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:22:30.127842 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.127815    2582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-installation-pull-secrets" pod "7da0156a-c68b-4e26-8344-852632f8c78e" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:22:30.127934 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.127901    2582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-image-registry-private-configuration" pod "7da0156a-c68b-4e26-8344-852632f8c78e" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:22:30.128146 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.128121    2582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-kube-api-access-6f772" pod "7da0156a-c68b-4e26-8344-852632f8c78e" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e"). InnerVolumeSpecName "kube-api-access-6f772". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:22:30.128240 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.128175    2582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-bound-sa-token" pod "7da0156a-c68b-4e26-8344-852632f8c78e" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:22:30.227351 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.227319    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-bound-sa-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:22:30.227351 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.227344    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-installation-pull-secrets\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:22:30.227351 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.227354    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-registry-certificates\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:22:30.227571 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.227363    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7da0156a-c68b-4e26-8344-852632f8c78e-trusted-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:22:30.227571 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.227374    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f772\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-kube-api-access-6f772\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:22:30.227571 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.227383    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7da0156a-c68b-4e26-8344-852632f8c78e-image-registry-private-configuration\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:22:30.428484 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.428407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:30.428484 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.428447    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:30.428662 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.428490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:30.429118 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.429089    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:30.430799 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.430780    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:30.430938 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.430916    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:30.529347 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.529308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:30.529514 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.529375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:30.529514 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.529399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:30.531846 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.531818    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:30.531846 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.531837    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:30.531978 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.531817    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:30.629808 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.629773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:22:30.629969 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.629823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:30.632142 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.632115    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:30.632247 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.632175    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"image-registry-569b8d479f-htb2l\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") " pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:22:30.678860 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.678790    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-d6zhm\""
Nov 01 12:22:30.680898 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.680880    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:30.684204 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.684189    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lt5kb\""
Nov 01 12:22:30.686101 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.686081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:30.726438 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.726408    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8q978\""
Nov 01 12:22:30.728556 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.728537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:30.731486 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.731408    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") pod \"7da0156a-c68b-4e26-8344-852632f8c78e\" (UID: \"7da0156a-c68b-4e26-8344-852632f8c78e\") "
Nov 01 12:22:30.733525 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.733466    2582 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls" pod "7da0156a-c68b-4e26-8344-852632f8c78e" (UID: "7da0156a-c68b-4e26-8344-852632f8c78e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:22:30.746152 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.746095    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kqn6l\""
Nov 01 12:22:30.752798 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.752551    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:30.770945 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.770920    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-m68cz\""
Nov 01 12:22:30.773152 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.773131    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:30.832660 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.832606    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7da0156a-c68b-4e26-8344-852632f8c78e-registry-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:22:30.860778 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.860751    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-62ffm\""
Nov 01 12:22:30.862836 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:30.862815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:31.016324 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.016294    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-569b8d479f-htb2l"
Nov 01 12:22:31.141672 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.141633    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79dd7d46d9-xfhjz"]
Nov 01 12:22:31.148741 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:22:31.148716    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426e64a6_ecf0_471e_b130_15f85716445f.slice/crio-6ec05f2a91ebf5c7608b4a8a7ca18f3bbc6681d0d1214e8a46d3ac231a4cd096 WatchSource:0}: Error finding container 6ec05f2a91ebf5c7608b4a8a7ca18f3bbc6681d0d1214e8a46d3ac231a4cd096: Status 404 returned error can't find the container with id 6ec05f2a91ebf5c7608b4a8a7ca18f3bbc6681d0d1214e8a46d3ac231a4cd096
Nov 01 12:22:31.163782 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.163757    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"]
Nov 01 12:22:31.167274 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.167248    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xm57l"]
Nov 01 12:22:31.210457 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.210435    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84fbdf67f8-6fmbg"]
Nov 01 12:22:31.220481 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:22:31.220451    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f50f8c_9c08_406f_aba6_83fc9a2f75fd.slice/crio-7257c348c8d0768131c2427c447d061a1673df34fc8c14462bee77f3837dd452 WatchSource:0}: Error finding container 7257c348c8d0768131c2427c447d061a1673df34fc8c14462bee77f3837dd452: Status 404 returned error can't find the container with id 7257c348c8d0768131c2427c447d061a1673df34fc8c14462bee77f3837dd452
Nov 01 12:22:31.221499 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.221359    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-744lb"]
Nov 01 12:22:31.230523 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:22:31.230497    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00596c9_ffa3_461b_81be_7315388deaf1.slice/crio-0e7ee8e5a10fc26da1e61184d229a320b3115097df9f7acd06ea25b505b0ae62 WatchSource:0}: Error finding container 0e7ee8e5a10fc26da1e61184d229a320b3115097df9f7acd06ea25b505b0ae62: Status 404 returned error can't find the container with id 0e7ee8e5a10fc26da1e61184d229a320b3115097df9f7acd06ea25b505b0ae62
Nov 01 12:22:31.243906 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.243868    2582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-569b8d479f-htb2l"]
Nov 01 12:22:31.247509 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.247451    2582 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-569b8d479f-htb2l"]
Nov 01 12:22:31.255817 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.255783    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"]
Nov 01 12:22:31.264602 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:22:31.264579    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3758a7d1_ccc2_4322_bd01_e742e5c11ff4.slice/crio-8c33d8154e2667ba4e26fcab11d1d2fb2034477f3f6d80b23e29b2a4e20b7ee2 WatchSource:0}: Error finding container 8c33d8154e2667ba4e26fcab11d1d2fb2034477f3f6d80b23e29b2a4e20b7ee2: Status 404 returned error can't find the container with id 8c33d8154e2667ba4e26fcab11d1d2fb2034477f3f6d80b23e29b2a4e20b7ee2
Nov 01 12:22:31.437110 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.437067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:31.485437 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.485407    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 12:22:31.489545 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.489517    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:31.538565 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.538483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:22:31.586759 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.586729    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 12:22:31.590867 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.590841    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:22:31.822900 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.822814    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-phch9\""
Nov 01 12:22:31.824913 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.824890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:31.860431 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.860395    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzkjs\""
Nov 01 12:22:31.862320 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:31.862295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:22:32.020539 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.020508    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn" event={"ID":"39facba4-434d-476e-9e84-a8388d78e4e9","Type":"ContainerStarted","Data":"8459c17c81a18ba53f4d923f2d486a46bfd55181679e028f9d18d4be6f0560d4"}
Nov 01 12:22:32.022011 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.021983    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz" event={"ID":"426e64a6-ecf0-471e-b130-15f85716445f","Type":"ContainerStarted","Data":"8470666bd588c28ab877fcb69f7dfd0672239bbe3aa8608d28fbbbb576b07d3f"}
Nov 01 12:22:32.022152 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.022026    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz" event={"ID":"426e64a6-ecf0-471e-b130-15f85716445f","Type":"ContainerStarted","Data":"6ec05f2a91ebf5c7608b4a8a7ca18f3bbc6681d0d1214e8a46d3ac231a4cd096"}
Nov 01 12:22:32.023145 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.023122    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xm57l" event={"ID":"ee4ddc2b-35a6-4862-b64c-bb137efcdd69","Type":"ContainerStarted","Data":"45ce23c20cce1a88d17abd4055a31855e94aa59683f96cee242f45f86d17818f"}
Nov 01 12:22:32.024206 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.024166    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml" event={"ID":"3758a7d1-ccc2-4322-bd01-e742e5c11ff4","Type":"ContainerStarted","Data":"8c33d8154e2667ba4e26fcab11d1d2fb2034477f3f6d80b23e29b2a4e20b7ee2"}
Nov 01 12:22:32.027232 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.027211    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg" event={"ID":"97f50f8c-9c08-406f-aba6-83fc9a2f75fd","Type":"ContainerStarted","Data":"538ebd31136705a064ccb990ba4e4140e920d6e27ec38000aabda94621b09c02"}
Nov 01 12:22:32.027333 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.027252    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg" event={"ID":"97f50f8c-9c08-406f-aba6-83fc9a2f75fd","Type":"ContainerStarted","Data":"7257c348c8d0768131c2427c447d061a1673df34fc8c14462bee77f3837dd452"}
Nov 01 12:22:32.027333 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.027278    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:32.029164 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.029146    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb" event={"ID":"c00596c9-ffa3-461b-81be-7315388deaf1","Type":"ContainerStarted","Data":"0e7ee8e5a10fc26da1e61184d229a320b3115097df9f7acd06ea25b505b0ae62"}
Nov 01 12:22:32.131911 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.131816    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz" podStartSLOduration=434.131793321 podStartE2EDuration="7m14.131793321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:32.130949807 +0000 UTC m=+66.081956204" watchObservedRunningTime="2026-11-01 12:22:32.131793321 +0000 UTC m=+66.082799701"
Nov 01 12:22:32.193523 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.193485    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7lhr"]
Nov 01 12:22:32.201656 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:22:32.201626    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370a3bfd_1661_4e44_b5d5_5f6b83db1f6c.slice/crio-76585ee8e6d28f78c7611868e223f47b5bd88670250b56909e45509680e609b3 WatchSource:0}: Error finding container 76585ee8e6d28f78c7611868e223f47b5bd88670250b56909e45509680e609b3: Status 404 returned error can't find the container with id 76585ee8e6d28f78c7611868e223f47b5bd88670250b56909e45509680e609b3
Nov 01 12:22:32.240969 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.240914    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg" podStartSLOduration=60.240899516 podStartE2EDuration="1m0.240899516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:32.240448222 +0000 UTC m=+66.191454610" watchObservedRunningTime="2026-11-01 12:22:32.240899516 +0000 UTC m=+66.191905896"
Nov 01 12:22:32.249083 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.248577    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p779s"]
Nov 01 12:22:32.257323 ip-10-0-138-76 kubenswrapper[2582]: W1101 12:22:32.257291    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3888b182_8fb4_49bd_9af3_873bcc3efa92.slice/crio-f9d52d8311b3c108adedc1b192c47785e342ab5d1aeafd2d8eea6d734fcfb0ac WatchSource:0}: Error finding container f9d52d8311b3c108adedc1b192c47785e342ab5d1aeafd2d8eea6d734fcfb0ac: Status 404 returned error can't find the container with id f9d52d8311b3c108adedc1b192c47785e342ab5d1aeafd2d8eea6d734fcfb0ac
Nov 01 12:22:32.653673 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.653632    2582 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7da0156a-c68b-4e26-8344-852632f8c78e" path="/var/lib/kubelet/pods/7da0156a-c68b-4e26-8344-852632f8c78e/volumes"
Nov 01 12:22:32.681814 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.681779    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:32.685417 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:32.685176    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:33.037500 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:33.037379    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7lhr" event={"ID":"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c","Type":"ContainerStarted","Data":"76585ee8e6d28f78c7611868e223f47b5bd88670250b56909e45509680e609b3"}
Nov 01 12:22:33.041173 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:33.041148    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p779s" event={"ID":"3888b182-8fb4-49bd-9af3-873bcc3efa92","Type":"ContainerStarted","Data":"f9d52d8311b3c108adedc1b192c47785e342ab5d1aeafd2d8eea6d734fcfb0ac"}
Nov 01 12:22:33.041957 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:33.041938    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:33.043375 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:33.043354    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:34.049881 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:34.049850    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb" event={"ID":"c00596c9-ffa3-461b-81be-7315388deaf1","Type":"ContainerStarted","Data":"4ca54e407cc63efa53b4ec641ad34726c2e4e5519b3418101c6b7d9b7677ed9e"}
Nov 01 12:22:34.158186 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:34.157313    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb" podStartSLOduration=396.783407986 podStartE2EDuration="6m39.157294354s" totalImagesPullingTime="2.373886368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:34.15620457 +0000 UTC m=+68.107210953" watchObservedRunningTime="2026-11-01 12:22:34.157294354 +0000 UTC m=+68.108300737"
Nov 01 12:22:35.054905 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.054857    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn" event={"ID":"39facba4-434d-476e-9e84-a8388d78e4e9","Type":"ContainerStarted","Data":"e2646ce993444aba594eaf9e727fcf529fadab1233ad049c1fbee248149eb651"}
Nov 01 12:22:35.055387 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.054933    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn" event={"ID":"39facba4-434d-476e-9e84-a8388d78e4e9","Type":"ContainerStarted","Data":"2ef1ba62c3ffeda93fc393377c8cf4c530efb66bc6d22c4c990756be43b22ede"}
Nov 01 12:22:35.057700 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.057668    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p779s" event={"ID":"3888b182-8fb4-49bd-9af3-873bcc3efa92","Type":"ContainerStarted","Data":"8cdbb2c1968501672528e30da4d4459d73a832a30b30d1a4240a8e38d186600f"}
Nov 01 12:22:35.057844 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.057733    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p779s" event={"ID":"3888b182-8fb4-49bd-9af3-873bcc3efa92","Type":"ContainerStarted","Data":"9649304992a0a536291b4937cbddf0b4d3074a4ac8d09de7b6d65df8a3abc37d"}
Nov 01 12:22:35.060088 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.060061    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xm57l" event={"ID":"ee4ddc2b-35a6-4862-b64c-bb137efcdd69","Type":"ContainerStarted","Data":"6991fc9df77d914b0bca858ed66070c23f2e1b783a4270b7aa2b7781212e1b2c"}
Nov 01 12:22:35.060245 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.060110    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xm57l" event={"ID":"ee4ddc2b-35a6-4862-b64c-bb137efcdd69","Type":"ContainerStarted","Data":"a225678cbf6b0cdec0f7be75faa22fdba71e938ac560cea8c7445bf5be3295b6"}
Nov 01 12:22:35.060245 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.060138    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:35.061778 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.061755    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml" event={"ID":"3758a7d1-ccc2-4322-bd01-e742e5c11ff4","Type":"ContainerStarted","Data":"e07c08e2cb8c8bc8453cfa402f953c1b788dc8093d2e1a2bc91faeab829bf0e1"}
Nov 01 12:22:35.162739 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.162682    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn" podStartSLOduration=430.049727402 podStartE2EDuration="7m13.162668979s" totalImagesPullingTime="3.112941577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:35.162204886 +0000 UTC m=+69.113211267" watchObservedRunningTime="2026-11-01 12:22:35.162668979 +0000 UTC m=+69.113675353"
Nov 01 12:22:35.271007 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.270226    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p779s" podStartSLOduration=67.173921742 podStartE2EDuration="1m9.270203527s" totalImagesPullingTime="2.096281785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:35.269406528 +0000 UTC m=+69.220412904" watchObservedRunningTime="2026-11-01 12:22:35.270203527 +0000 UTC m=+69.221209902"
Nov 01 12:22:35.380616 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.380509    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml" podStartSLOduration=427.287823365 podStartE2EDuration="7m10.380490166s" totalImagesPullingTime="3.092666801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:35.379673844 +0000 UTC m=+69.330680229" watchObservedRunningTime="2026-11-01 12:22:35.380490166 +0000 UTC m=+69.331496548"
Nov 01 12:22:35.488903 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:35.488845    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xm57l" podStartSLOduration=34.553119851 podStartE2EDuration="37.488829227s" totalImagesPullingTime="2.935709376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:35.487924052 +0000 UTC m=+69.438930430" watchObservedRunningTime="2026-11-01 12:22:35.488829227 +0000 UTC m=+69.439835607"
Nov 01 12:22:37.932755 ip-10-0-138-76 kubenswrapper[2582]: I1101 12:22:37.932721    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:22:38.415316 ip-10-0-138-76 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 12:22:38.427483 ip-10-0-138-76 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 12:22:38.427651 ip-10-0-138-76 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 12:22:38.427704 ip-10-0-138-76 systemd[1]: kubelet.service: Consumed 4.371s CPU time.
Nov 01 12:22:38.437584 ip-10-0-138-76 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 12:22:38.521509 ip-10-0-138-76 kubenswrapper[5725]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:22:38.521509 ip-10-0-138-76 kubenswrapper[5725]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 12:22:38.521509 ip-10-0-138-76 kubenswrapper[5725]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:22:38.521509 ip-10-0-138-76 kubenswrapper[5725]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:22:38.523627 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523541    5725 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:38.523627 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523627    5725 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:38.523627 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523632    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523635    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523639    5725 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523641    5725 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523644    5725 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523647    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523649    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523652    5725 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523655    5725 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523657    5725 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523660    5725 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523663    5725 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523666    5725 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523668    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523671    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523674    5725 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523676    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523679    5725 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523682    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:38.523725 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523684    5725 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523687    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523690    5725 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523698    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523701    5725 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523704    5725 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523706    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523709    5725 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523712    5725 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523715    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523717    5725 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523720    5725 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523723    5725 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523725    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523729    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523732    5725 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523734    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523737    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523740    5725 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523743    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:38.524219 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523746    5725 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523749    5725 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523751    5725 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523754    5725 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523756    5725 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523759    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523762    5725 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523764    5725 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523767    5725 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523769    5725 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523772    5725 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523774    5725 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523777    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523780    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523783    5725 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523785    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523789    5725 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523792    5725 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523794    5725 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523799    5725 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:38.524703 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523804    5725 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523807    5725 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523811    5725 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523814    5725 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523816    5725 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523819    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523822    5725 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523832    5725 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523835    5725 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523838    5725 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523840    5725 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523843    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523846    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523850    5725 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523853    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523856    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523859    5725 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523863    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523866    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:38.525204 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523868    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523871    5725 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523874    5725 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523876    5725 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523879    5725 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523882    5725 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523885    5725 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523887    5725 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523890    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523892    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523896    5725 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523899    5725 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523901    5725 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523904    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523907    5725 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523909    5725 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523912    5725 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523914    5725 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523917    5725 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523920    5725 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:38.525669 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523922    5725 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523925    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523927    5725 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523929    5725 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523932    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.523935    5725 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524662    5725 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524669    5725 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524672    5725 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524675    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524678    5725 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524681    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524684    5725 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524686    5725 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524690    5725 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524692    5725 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524695    5725 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524698    5725 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524702    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:38.526159 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524705    5725 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524708    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524711    5725 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524714    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524717    5725 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524720    5725 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524722    5725 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524725    5725 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524727    5725 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524730    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524733    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524735    5725 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524738    5725 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524740    5725 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524743    5725 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524746    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524749    5725 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524751    5725 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524754    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524757    5725 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:38.526723 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524759    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524762    5725 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524765    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524767    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524770    5725 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524774    5725 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524777    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524780    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524783    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524786    5725 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524788    5725 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524791    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524795    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524797    5725 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524800    5725 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524803    5725 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524805    5725 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524809    5725 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524811    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:38.527505 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524814    5725 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524817    5725 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524819    5725 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524822    5725 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524824    5725 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524827    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524829    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524832    5725 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524835    5725 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524837    5725 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524840    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524842    5725 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524846    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524848    5725 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524851    5725 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524855    5725 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524858    5725 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524862    5725 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524865    5725 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524868    5725 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:38.528022 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524871    5725 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524874    5725 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524876    5725 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524879    5725 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524882    5725 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524885    5725 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524887    5725 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524890    5725 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524893    5725 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524895    5725 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524898    5725 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524901    5725 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524903    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524906    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524909    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524911    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524914    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524916    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524919    5725 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:38.528527 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524922    5725 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524924    5725 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524927    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524929    5725 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524932    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524934    5725 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524937    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524939    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524942    5725 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524944    5725 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524947    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524950    5725 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524953    5725 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524956    5725 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.524958    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525469    5725 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525476    5725 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525479    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525482    5725 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525485    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:38.528980 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525487    5725 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525490    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525493    5725 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525495    5725 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525498    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525501    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525504    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525506    5725 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525509    5725 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525511    5725 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525514    5725 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525516    5725 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525519    5725 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525521    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525524    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525526    5725 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525531    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525534    5725 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525537    5725 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525539    5725 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:38.529486 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525542    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525545    5725 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525547    5725 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525550    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525552    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525555    5725 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525558    5725 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525561    5725 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525564    5725 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525566    5725 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525569    5725 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525571    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525574    5725 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525576    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525579    5725 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525581    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525584    5725 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525587    5725 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525590    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525593    5725 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:38.529970 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525595    5725 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525598    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525601    5725 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525603    5725 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525607    5725 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525611    5725 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525614    5725 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525617    5725 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525624    5725 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525627    5725 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525629    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525633    5725 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525635    5725 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525638    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525641    5725 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525643    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525646    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525649    5725 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525651    5725 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:38.530467 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525654    5725 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525656    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525659    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525662    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525664    5725 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525667    5725 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525670    5725 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525672    5725 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525675    5725 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525677    5725 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525680    5725 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525683    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525686    5725 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525689    5725 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525691    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525694    5725 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525696    5725 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525699    5725 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525701    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525704    5725 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:38.530928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525706    5725 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525710    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525713    5725 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525715    5725 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525719    5725 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525722    5725 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525726    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525728    5725 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525731    5725 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525734    5725 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525737    5725 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525740    5725 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525743    5725 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525746    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525748    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525751    5725 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525753    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525756    5725 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525758    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:38.531419 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525761    5725 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525765    5725 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.525767    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525856    5725 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525864    5725 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525901    5725 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525907    5725 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525911    5725 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525915    5725 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525919    5725 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525923    5725 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525926    5725 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525929    5725 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525933    5725 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525938    5725 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525942    5725 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525945    5725 flags.go:64] FLAG: --cgroup-root=""
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525948    5725 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525951    5725 flags.go:64] FLAG: --client-ca-file=""
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525954    5725 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525957    5725 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525960    5725 flags.go:64] FLAG: --cluster-domain=""
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525963    5725 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525966    5725 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 12:22:38.531882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525969    5725 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525973    5725 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525977    5725 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525980    5725 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525983    5725 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525986    5725 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525989    5725 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525992    5725 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525995    5725 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.525999    5725 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526002    5725 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526007    5725 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526010    5725 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526013    5725 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526017    5725 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526021    5725 flags.go:64] FLAG: --enable-server="true"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526023    5725 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526028    5725 flags.go:64] FLAG: --event-burst="100"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526031    5725 flags.go:64] FLAG: --event-qps="50"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526034    5725 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526037    5725 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526040    5725 flags.go:64] FLAG: --eviction-hard=""
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526058    5725 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526061    5725 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526065    5725 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 12:22:38.532474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526068    5725 flags.go:64] FLAG: --eviction-soft=""
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526071    5725 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526074    5725 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526077    5725 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526080    5725 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526083    5725 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526086    5725 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526089    5725 flags.go:64] FLAG: --feature-gates=""
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526094    5725 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526097    5725 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526100    5725 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526103    5725 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526106    5725 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526110    5725 flags.go:64] FLAG: --help="false"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526113    5725 flags.go:64] FLAG: --hostname-override="ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526116    5725 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526119    5725 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526122    5725 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526127    5725 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526130    5725 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526133    5725 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526136    5725 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526140    5725 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 12:22:38.533142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526143    5725 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526146    5725 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526149    5725 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526152    5725 flags.go:64] FLAG: --kube-reserved=""
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526155    5725 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526159    5725 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526164    5725 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526166    5725 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526169    5725 flags.go:64] FLAG: --lock-file=""
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526172    5725 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526175    5725 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526178    5725 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526184    5725 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526187    5725 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526190    5725 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526193    5725 flags.go:64] FLAG: --logging-format="text"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526196    5725 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526199    5725 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526202    5725 flags.go:64] FLAG: --manifest-url=""
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526205    5725 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526210    5725 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526213    5725 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526216    5725 flags.go:64] FLAG: --max-pods="110"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526220    5725 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526223    5725 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 12:22:38.533749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526226    5725 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526229    5725 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526234    5725 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526237    5725 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526239    5725 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526246    5725 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526249    5725 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526256    5725 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526259    5725 flags.go:64] FLAG: --pod-cidr=""
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526262    5725 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526265    5725 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526268    5725 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526271    5725 flags.go:64] FLAG: --port="10250"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526276    5725 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526278    5725 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ef7d165ce2cbabee"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526282    5725 flags.go:64] FLAG: --qos-reserved=""
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526285    5725 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526288    5725 flags.go:64] FLAG: --register-node="true"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526291    5725 flags.go:64] FLAG: --register-with-taints=""
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526295    5725 flags.go:64] FLAG: --registry-burst="10"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526298    5725 flags.go:64] FLAG: --registry-qps="5"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526301    5725 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526304    5725 flags.go:64] FLAG: --reserved-memory=""
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526308    5725 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526311    5725 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526314    5725 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 12:22:38.534353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526317    5725 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526320    5725 flags.go:64] FLAG: --runonce="false"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526323    5725 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526326    5725 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526329    5725 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526333    5725 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526336    5725 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526339    5725 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526343    5725 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526347    5725 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526350    5725 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526353    5725 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526356    5725 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526358    5725 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526362    5725 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526365    5725 flags.go:64] FLAG: --system-cgroups=""
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526368    5725 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526450    5725 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526503    5725 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526525    5725 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526536    5725 flags.go:64] FLAG: --tls-min-version=""
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526542    5725 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526547    5725 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526553    5725 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526560    5725 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 12:22:38.535236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526566    5725 flags.go:64] FLAG: --v="2"
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526600    5725 flags.go:64] FLAG: --version="false"
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526607    5725 flags.go:64] FLAG: --vmodule=""
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526613    5725 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.526620    5725 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527654    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527669    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527676    5725 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527679    5725 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527683    5725 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527685    5725 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527688    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527691    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527694    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527696    5725 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527700    5725 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527703    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527705    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527708    5725 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527711    5725 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527713    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:38.535840 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527718    5725 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527721    5725 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527725    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527727    5725 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527730    5725 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527733    5725 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527736    5725 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527738    5725 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527741    5725 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527744    5725 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527747    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527750    5725 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527753    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527756    5725 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527759    5725 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527761    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527765    5725 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527768    5725 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527771    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:38.536370 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527774    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527777    5725 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527781    5725 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527784    5725 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527786    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527789    5725 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527792    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527794    5725 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527797    5725 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527800    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527802    5725 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527806    5725 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527810    5725 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527813    5725 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527816    5725 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527819    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527822    5725 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527833    5725 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527836    5725 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:38.536888 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527839    5725 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527841    5725 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527844    5725 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527847    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527850    5725 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527853    5725 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527856    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527859    5725 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527862    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527865    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527868    5725 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527871    5725 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527873    5725 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527876    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527879    5725 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527881    5725 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527884    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527887    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527890    5725 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:38.537694 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527893    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527898    5725 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527902    5725 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527906    5725 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527909    5725 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527912    5725 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527914    5725 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527917    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527920    5725 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527923    5725 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527926    5725 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527928    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527932    5725 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527934    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527937    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527940    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527943    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527946    5725 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527948    5725 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527951    5725 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:38.538496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527954    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527956    5725 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527959    5725 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527962    5725 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527964    5725 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527967    5725 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527969    5725 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527972    5725 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527974    5725 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527977    5725 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527980    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527983    5725 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.527985    5725 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.528018    5725 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528100    5725 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528106    5725 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:38.539067 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528109    5725 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528112    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528115    5725 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528117    5725 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528121    5725 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528123    5725 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528126    5725 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528128    5725 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528131    5725 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528134    5725 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528137    5725 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528140    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528142    5725 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528145    5725 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528147    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528150    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528153    5725 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528155    5725 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528158    5725 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528160    5725 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:38.539496 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528163    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528165    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528168    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528170    5725 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528174    5725 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528177    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528179    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528182    5725 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528185    5725 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528187    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528190    5725 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528193    5725 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528196    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528198    5725 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528201    5725 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528203    5725 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528206    5725 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528209    5725 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528211    5725 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:38.540014 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528214    5725 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528222    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528225    5725 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528228    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528231    5725 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528233    5725 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528236    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528238    5725 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528241    5725 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528244    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528246    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528249    5725 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528251    5725 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528254    5725 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528256    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528259    5725 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528261    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528264    5725 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528267    5725 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:38.540586 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528269    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528272    5725 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528276    5725 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528280    5725 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528283    5725 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528286    5725 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528288    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528291    5725 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528293    5725 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528296    5725 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528298    5725 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528301    5725 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528303    5725 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528306    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528309    5725 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528312    5725 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528316    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528318    5725 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528321    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:38.541400 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528324    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528326    5725 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528329    5725 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528332    5725 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528334    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528337    5725 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528339    5725 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528342    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528345    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528347    5725 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528349    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528353    5725 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528357    5725 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528361    5725 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528363    5725 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528366    5725 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528370    5725 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528373    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528375    5725 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:38.542172 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528378    5725 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528380    5725 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528383    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528385    5725 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528389    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528391    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528396    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.528398    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.532096    5725 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.532113    5725 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.532131    5725 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.532136    5725 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532194    5725 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532200    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532203    5725 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532207    5725 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532210    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532213    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532216    5725 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532219    5725 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:38.542973 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532221    5725 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532224    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532227    5725 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532230    5725 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532233    5725 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532236    5725 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532239    5725 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532241    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532244    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532247    5725 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532250    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532253    5725 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532255    5725 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532258    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532261    5725 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532264    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532266    5725 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532269    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532272    5725 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532275    5725 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:38.543817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532277    5725 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532281    5725 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532284    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532287    5725 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532289    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532292    5725 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532295    5725 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532297    5725 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532300    5725 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532302    5725 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532305    5725 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532308    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532311    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532314    5725 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532317    5725 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532320    5725 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532324    5725 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532328    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:38.544727 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532332    5725 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532335    5725 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532338    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532342    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532345    5725 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532348    5725 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532350    5725 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532353    5725 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532355    5725 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532358    5725 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532360    5725 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532363    5725 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532365    5725 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532368    5725 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532370    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532373    5725 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532376    5725 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532379    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532381    5725 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532384    5725 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:38.545632 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532386    5725 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532389    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532391    5725 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532394    5725 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532396    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532399    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532401    5725 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532404    5725 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532407    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532409    5725 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532411    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532414    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532418    5725 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532422    5725 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532425    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532428    5725 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532431    5725 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532434    5725 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532437    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:38.546292 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532440    5725 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532443    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532445    5725 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532447    5725 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532450    5725 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532453    5725 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532455    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532458    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532460    5725 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532462    5725 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532466    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532468    5725 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532471    5725 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532473    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532476    5725 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532479    5725 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532481    5725 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532484    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532487    5725 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532490    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:38.546747 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532492    5725 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.532521    5725 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532668    5725 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532673    5725 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532676    5725 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532680    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532683    5725 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532685    5725 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532688    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532690    5725 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532693    5725 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532696    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532698    5725 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532701    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532703    5725 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:38.547298 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532707    5725 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532712    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532714    5725 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532717    5725 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532720    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532722    5725 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532725    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532727    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532730    5725 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532733    5725 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532735    5725 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532738    5725 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532741    5725 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532743    5725 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532746    5725 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532749    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532752    5725 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532754    5725 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532757    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532760    5725 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:38.547660 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532762    5725 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532765    5725 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532767    5725 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532770    5725 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532772    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532775    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532778    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532780    5725 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532783    5725 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532785    5725 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532788    5725 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532790    5725 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532793    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532795    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532798    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532801    5725 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532803    5725 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532806    5725 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532809    5725 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532811    5725 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:38.548166 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532813    5725 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532836    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532840    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532843    5725 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532846    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532849    5725 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532852    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532854    5725 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532857    5725 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532861    5725 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532864    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532867    5725 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532870    5725 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532873    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532876    5725 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532878    5725 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532881    5725 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532884    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532887    5725 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:38.548683 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532889    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532892    5725 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532894    5725 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532897    5725 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532900    5725 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532903    5725 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532906    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532908    5725 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532911    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532913    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532916    5725 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532918    5725 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532921    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532923    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532926    5725 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532929    5725 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532932    5725 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532934    5725 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532937    5725 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:38.549157 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532940    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532942    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532945    5725 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532948    5725 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532950    5725 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532953    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532955    5725 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532958    5725 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532960    5725 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532963    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532965    5725 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532968    5725 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532971    5725 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532974    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.532976    5725 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.532998    5725 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:22:38.549633 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533030    5725 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533034    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533037    5725 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533040    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533042    5725 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533045    5725 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533063    5725 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533068    5725 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533072    5725 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533075    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533078    5725 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533080    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533083    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533087    5725 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533090    5725 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533092    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533095    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533097    5725 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533100    5725 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:38.550024 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533103    5725 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533105    5725 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533108    5725 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533111    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533113    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533117    5725 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533119    5725 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533122    5725 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533124    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533127    5725 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533129    5725 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533132    5725 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533134    5725 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533137    5725 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533139    5725 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533142    5725 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533145    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533147    5725 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533150    5725 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:38.550500 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533153    5725 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533156    5725 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533158    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533161    5725 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533164    5725 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533166    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533169    5725 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533171    5725 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533174    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533177    5725 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533180    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533182    5725 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533191    5725 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533195    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533198    5725 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533200    5725 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533205    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533208    5725 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533211    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:38.550956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533213    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533216    5725 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533218    5725 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533221    5725 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533223    5725 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533226    5725 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533229    5725 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533231    5725 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533234    5725 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533237    5725 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533239    5725 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533243    5725 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533245    5725 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533248    5725 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533251    5725 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533253    5725 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533256    5725 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533259    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533261    5725 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533264    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:38.551548 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533266    5725 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533269    5725 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533272    5725 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533274    5725 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533277    5725 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533282    5725 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533284    5725 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533287    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533290    5725 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533293    5725 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533296    5725 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533299    5725 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533302    5725 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533304    5725 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533307    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533309    5725 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533312    5725 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533315    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533317    5725 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533320    5725 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:38.552104 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533322    5725 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533325    5725 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533329    5725 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533332    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533335    5725 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533340    5725 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533343    5725 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533345    5725 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:38.533348    5725 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.533403    5725 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.534386    5725 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.534450    5725 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.534622    5725 server.go:1014] "Starting client certificate rotation"
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.534756    5725 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.534799    5725 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 12:16:26 +0000 UTC" deadline="2028-07-09 17:01:29.822594398 +0000 UTC"
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.534857    5725 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14788h38m51.287739887s"
Nov 01 12:22:38.552600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.535078    5725 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 12:22:38.552990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.536304    5725 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 12:22:38.552990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.538037    5725 log.go:25] "Validated CRI v1 runtime API"
Nov 01 12:22:38.552990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.540155    5725 log.go:25] "Validated CRI v1 image API"
Nov 01 12:22:38.552990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.540878    5725 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 12:22:38.552990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.545196    5725 fs.go:135] Filesystem UUIDs: map[0ba2cb13-acb0-4790-9d5a-d8d737293e55:/dev/nvme0n1p3 30bfab0c-9e90-4341-a0e9-95f92e235026:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 01 12:22:38.553211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.545258    5725 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0215e6a2e22be96d6ed38a4f26830d0cf4fa8ec7763300c0e4b01dcdf876e710/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0215e6a2e22be96d6ed38a4f26830d0cf4fa8ec7763300c0e4b01dcdf876e710/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e7ee8e5a10fc26da1e61184d229a320b3115097df9f7acd06ea25b505b0ae62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e7ee8e5a10fc26da1e61184d229a320b3115097df9f7acd06ea25b505b0ae62/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16839b5525d977e2ceb2cf35c97f94ddccca438d46194fac581c29d9c5798dab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16839b5525d977e2ceb2cf35c97f94ddccca438d46194fac581c29d9c5798dab/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ec4e303cde0fe00ebe4cf59d0ff5ef5eaa4808adeda24c467573b2b44bbe584/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ec4e303cde0fe00ebe4cf59d0ff5ef5eaa4808adeda24c467573b2b44bbe584/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28e2f6de099644659469607c349c0bd5570004e29c98a262e6f239abbf7e0a44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28e2f6de099644659469607c349c0bd5570004e29c98a262e6f239abbf7e0a44/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a18ed8bb12322dbc7c4ee48db29d33cf28a4e5956e5f20f4908ddea6af352e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a18ed8bb12322dbc7c4ee48db29d33cf28a4e5956e5f20f4908ddea6af352e3/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36ea08297cbe07593735aaf7420137f4a1f2f8e85422bda88076bd722c990946/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36ea08297cbe07593735aaf7420137f4a1f2f8e85422bda88076bd722c990946/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45ce23c20cce1a88d17abd4055a31855e94aa59683f96cee242f45f86d17818f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45ce23c20cce1a88d17abd4055a31855e94aa59683f96cee242f45f86d17818f/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cef24fe79149cdd7d5c18280de7fe567075cd1a6ecec869b74f34b5f17fe665/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cef24fe79149cdd7d5c18280de7fe567075cd1a6ecec869b74f34b5f17fe665/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/524fba8833ffac4a77f7472fa799f01603d6041c70eae7eaf6278c512c9c9dd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/524fba8833ffac4a77f7472fa799f01603d6041c70eae7eaf6278c512c9c9dd1/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/533dfbadc7dd1c10cf42b78f0108ef1a2fb34aa2d1c1af95ef5fd5340cf3dbb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/533dfbadc7dd1c10cf42b78f0108ef1a2fb34aa2d1c1af95ef5fd5340cf3dbb5/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f6cfad4d2a416533c267b61ca1f75a9dbbf3e8efd36ad921a71a3591c5177cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f6cfad4d2a416533c267b61ca1f75a9dbbf3e8efd36ad921a71a3591c5177cf/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fd19072fe78d3948dfeddfb760348f6ba8d08f7ac323b3839980b06cf5884f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fd19072fe78d3948dfeddfb760348f6ba8d08f7ac323b3839980b06cf5884f5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62e8634a9944589861c55c9744227e50c8ca411f4566a85ef8cd3802fb43bc0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62e8634a9944589861c55c9744227e50c8ca411f4566a85ef8cd3802fb43bc0f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c9208b0b436be94df86d2bb4a955ecb04f0ac2ad90da7afa1e23e96b3d1e459/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c9208b0b436be94df86d2bb4a955ecb04f0ac2ad90da7afa1e23e96b3d1e459/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ec05f2a91ebf5c7608b4a8a7ca18f3bbc6681d0d1214e8a46d3ac231a4cd096/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ec05f2a91ebf5c7608b4a8a7ca18f3bbc6681d0d1214e8a46d3ac231a4cd096/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7257c348c8d0768131c2427c447d061a1673df34fc8c14462bee77f3837dd452/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7257c348c8d0768131c2427c447d061a1673df34fc8c14462bee77f3837dd452/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76585ee8e6d28f78c7611868e223f47b5bd88670250b56909e45509680e609b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76585ee8e6d28f78c7611868e223f47b5bd88670250b56909e45509680e609b3/userdata/shm major:0 minor:410 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b1b9c0c9cc7dd8e7d04dd704ffc64bffc986a97bac42c4a6a8835321eb59e18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b1b9c0c9cc7dd8e7d04dd704ffc64bffc986a97bac42c4a6a8835321eb59e18/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80e025ea2ea4cacbf74cce32958967710c1fd4d417b205cb290b96e28946efc1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80e025ea2ea4cacbf74cce32958967710c1fd4d417b205cb290b96e28946efc1/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8100acd04afd92612cb431b9ebf075b8d2d6d5de959dd4276468bf44fe4d78b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8100acd04afd92612cb431b9ebf075b8d2d6d5de959dd4276468bf44fe4d78b0/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81fffd4e6988fb157febb30c1ca3844c26047f5cecb478327b88b6eb667108da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81fffd4e6988fb157febb30c1ca3844c26047f5cecb478327b88b6eb667108da/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8459c17c81a18ba53f4d923f2d486a46bfd55181679e028f9d18d4be6f0560d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8459c17c81a18ba53f4d923f2d486a46bfd55181679e028f9d18d4be6f0560d4/userdata/shm major:0 minor:368 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c33d8154e2667ba4e26fcab11d1d2fb2034477f3f6d80b23e29b2a4e20b7ee2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c33d8154e2667ba4e26fcab11d1d2fb2034477f3f6d80b23e29b2a4e20b7ee2/userdata/shm major:0 minor:378 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a024a7bf0ef503a9b7d0b9203af8167c404ea1e5cbcba3cd270ba730a79a3d14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a024a7bf0ef503a9b7d0b9203af8167c404ea1e5cbcba3cd270ba730a79a3d14/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8dd251ce1d9c2ca22b64a503d4545c365aa77e777504adfae1d628ce90e6b64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8dd251ce1d9c2ca22b64a503d4545c365aa77e777504adfae1d628ce90e6b64/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbe766f5bec5c46aaa06e6ea37b0424e2f12dd74f5865c0d225e4e6df89050d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbe766f5bec5c46aaa06e6ea37b0424e2f12dd74f5865c0d225e4e6df89050d8/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d176df9d439e171136a90a031eccdffbe8063a47d766bf548753b7e098f396fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d176df9d439e171136a90a031eccdffbe8063a47d766bf548753b7e098f396fc/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e90a2419d0c439d98c6bbe650e374cc26457eaf6b79fb89f8d1bf90dc041032f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e90a2419d0c439d98c6bbe650e374cc26457eaf6b79fb89f8d1bf90dc041032f/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3d41b277973477bd26c72f0c47fbb5091275143533952735ede44b700b6ee37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3d41b277973477bd26c72f0c47fbb5091275143533952735ede44b700b6ee37/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9d52d8311b3c108adedc1b192c47785e342ab5d1aeafd2d8eea6d734fcfb0ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9d52d8311b3c108adedc1b192c47785e342ab5d1aeafd2d8eea6d734fcfb0ac/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03ef7e88-664f-413b-9424-c16a88d3b58c/volumes/kubernetes.io~projected/kube-api-access-hhrg8:{mountpoint:/var/lib/kubelet/pods/03ef7e88-664f-413b-9424-c16a88d3b58c/volumes/kubernetes.io~projected/kube-api-access-hhrg8 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03ef7e88-664f-413b-9424-c16a88d3b58c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/03ef7e88-664f-413b-9424-c16a88d3b58c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19e0e140-91e4-4d92-b1ac-0ea2115e3467/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/19e0e140-91e4-4d92-b1ac-0ea2115e3467/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19e0e140-91e4-4d92-b1ac-0ea2115e3467/volumes/kubernetes.io~projected/kube-api-access-l4rtn:{mountpoint:/var/lib/kubelet/pods/19e0e140-91e4-4d92-b1ac-0ea2115e3467/volumes/kubernetes.io~projected/kube-api-access-l4rtn major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19e0e140-91e4-4d92-b1ac-0ea2115e3467/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/19e0e140-91e4-4d92-b1ac-0ea2115e3467/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34c9bd1f-837b-4c0c-83da-3e5b09856707/volumes/kubernetes.io~projected/kube-api-access-6bkxh:{mountpoint:/var/lib/kubelet/pods/34c9bd1f-837b-4c0c-83da-3e5b09856707/volumes/kubernetes.io~projected/kube-api-access-6bkxh major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34c9bd1f-837b-4c0c-83da-3e5b09856707/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/34c9bd1f-837b-4c0c-83da-3e5b09856707/volumes/kubernetes.io~secret/cert major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3758a7d1-ccc2-4322-bd01-e742e5c11ff4/volumes/kubernetes.io~projected/kube-api-access-kzvmj:{mountpoint:/var/lib/kubelet/pods/3758a7d1-ccc2-4322-bd01-e742e5c11ff4/volumes/kubernetes.io~projected/kube-api-access-kzvmj major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3758a7d1-ccc2-4322-bd01-e742e5c11ff4/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/3758a7d1-ccc2-4322-bd01-e742e5c11ff4/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3888b182-8fb4-49bd-9af3-873bcc3efa92/volumes/kubernetes.io~projected/kube-api-access-cqbs7:{mountpoint:/var/lib/kubelet/pods/3888b182-8fb4-49bd-9af3-873bcc3efa92/volumes/kubernetes.io~projected/kube-api-access-cqbs7 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3888b182-8fb4-49bd-9af3-873bcc3efa92/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3888b182-8fb4-49bd-9af3-873bcc3efa92/volumes/kubernetes.io~secret/metrics-certs major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3953b512-91d0-401c-8397-0b9314e7a87b/volumes/kubernetes.io~projected/kube-api-access-rk8pw:{mountpoint:/var/lib/kubelet/pods/3953b512-91d0-401c-8397-0b9314e7a87b/volumes/kubernetes.io~projected/kube-api-access-rk8pw major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39facba4-434d-476e-9e84-a8388d78e4e9/volumes/kubernetes.io~projected/kube-api-access-4s67j:{mountpoint:/var/lib/kubelet/pods/39facba4-434d-476e-9e84-a8388d78e4e9/volumes/kubernetes.io~projected/kube-api-access-4s67j major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39facba4-434d-476e-9e84-a8388d78e4e9/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/39facba4-434d-476e-9e84-a8388d78e4e9/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843/volumes/kubernetes.io~projected/kube-api-access-x2rjb:{mountpoint:/var/lib/kubelet/pods/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843/volumes/kubernetes.io~projected/kube-api-access-x2rjb major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf/volumes/kubernetes.io~projected/kube-api-access-6j6dv:{mountpoint:/var/lib/kubelet/pods/3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf/volumes/kubernetes.io~projected/kube-api-access-6j6dv major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~projected/kube-api-access-pqv4d:{mountpoint:/var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~projected/kube-api-access-pqv4d major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46433966-ea95-45b6-9092-c2f73bd7cd53/volumes/kubernetes.io~projected/kube-api-access-qlch9:{mountpoint:/var/lib/kubelet/pods/46433966-ea95-45b6-9092-c2f73bd7cd53/volumes/kubernetes.io~projected/kube-api-access-qlch9 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47f53f35-b29e-4db5-91c8-0df05af362e7/volumes/kubernetes.io~projected/kube-api-access-x26dv:{mountpoint:/var/lib/kubelet/pods/47f53f35-b29e-4db5-91c8-0df05af362e7/volumes/kubernetes.io~projected/kube-api-access-x26dv major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cf38c23-496f-4661-85ce-586416959d95/volumes/kubernetes.io~projected/kube-api-access-jsjtg:{mountpoint:/var/lib/kubelet/pods/4cf38c23-496f-4661-85ce-586416959d95/volumes/kubernetes.io~projected/kube-api-access-jsjtg major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cedf131-a296-4792-8328-f3ad030d3003/volumes/kubernetes.io~projected/kube-api-access-7bmhd:{mountpoint:/var/lib/kubelet/pods/5cedf131-a296-4792-8328-f3ad030d3003/volumes/kubernetes.io~projected/kube-api-access-7bmhd major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/690056c7-6829-4caf-b59e-e2650d20bb47/volumes/kubernetes.io~projected/kube-api-access-54rjm:{mountpoint:/var/lib/kubelet/pods/690056c7-6829-4caf-b59e-e2650d20bb47/volumes/kubernetes.io~projected/kube-api-access-54rjm major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/693b0401-a9f9-41a4-b2e8-d3e3d6f37477/volumes/kubernetes.io~projected/kube-api-access-5gs79:{mountpoint:/var/lib/kubelet/pods/693b0401-a9f9-41a4-b2e8-d3e3d6f37477/volumes/kubernetes.io~projected/kube-api-access-5gs79 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36/volumes/kubernetes.io~empty-dir/tmp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36/volumes/kubernetes.io~projected/kube-api-access-zwsm4:{mountpoint:/var/lib/kubelet/pods/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36/volumes/kubernetes.io~projected/kube-api-access-zwsm4 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~projected/kube-api-access-9wr4q:{mountpoint:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~projected/kube-api-access-9wr4q major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~projected/registry-tls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b41f9fc-afce-464d-a18c-330694f80140/volumes/kubernetes.io~projected/kube-api-access-6rjsj:{mountpoint:/var/lib/kubelet/pods/9b41f9fc-afce-464d-a18c-330694f80140/volumes/kubernetes.io~projected/kube-api-access-6rjsj major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b41f9fc-afce-464d-a18c-330694f80140/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9b41f9fc-afce-464d-a18c-330694f80140/volumes/kubernetes.io~secret/serving-cert major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c/volumes/kubernetes.io~projected/kube-api-access-h7csw:{mountpoint:/var/lib/kubelet/pods/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c/volumes/kubernetes.io~projected/kube-api-access-h7csw major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6cb3539-d27f-4954-9b4d-e730dfd5809d/volumes/kubernetes.io~projected/kube-api-access-qhf84:{mountpoint:/var/lib/kubelet/pods/b6cb3539-d27f-4954-9b4d-e730dfd5809d/volumes/kubernetes.io~projected/kube-api-access-qhf84 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bad6344f-4c79-4140-abcc-2c4ae3d63049/volumes/kubernetes.io~projected/kube-api-access-vc6kk:{mountpoint:/var/lib/kubelet/pods/bad6344f-4c79-4140-abcc-2c4ae3d63049/volumes/kubernetes.io~projected/kube-api-access-vc6kk major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c00596c9-ffa3-461b-81be-7315388deaf1/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/c00596c9-ffa3-461b-81be-7315388deaf1/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c09c0770-6cec-4827-9fb0-b07464653b27/volumes/kubernetes.io~projected/kube-api-access-fkkmg:{mountpoint:/var/lib/kubelet/pods/c09c0770-6cec-4827-9fb0-b07464653b27/volumes/kubernetes.io~projected/kube-api-access-fkkmg major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1efc2ea-a886-4c13-97a7-b1ddbd621feb/volumes/kubernetes.io~projected/kube-api-access-m8lp6:{mountpoint:/var/lib/kubelet/pods/d1efc2ea-a886-4c13-97a7-b1ddbd621feb/volumes/kubernetes.io~projected/kube-api-access-m8lp6 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1efc2ea-a886-4c13-97a7-b1ddbd621feb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d1efc2ea-a886-4c13-97a7-b1ddbd621feb/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5291295-5fd1-4e6e-b52b-c940c088168d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e5291295-5fd1-4e6e-b52b-c940c088168d/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4ddc2b-35a6-4862-b64c-bb137efcdd69/volumes/kubernetes.io~projected/kube-api-access-4mxw6:{mountpoint:/var/lib/kubelet/pods/ee4ddc2b-35a6-4862-b64c-bb137efcdd69/volumes/kubernetes.io~projected/kube-api-access-4mxw6 major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4ddc2b-35a6-4862-b64c-bb137efcdd69/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/ee4ddc2b-35a6-4862-b64c-bb137efcdd69/volumes/kubernetes.io~secret/metrics-tls major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b43ebe-6a94-45f6-a613-0931f4607e9e/volumes/kubernetes.io~projected/kube-api-access-4t59b:{mountpoint:/var/lib/kubelet/pods/f8b43ebe-6a94-45f6-a613-0931f4607e9e/volumes/kubernetes.io~projected/kube-api-access-4t59b major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2/volumes/kubernetes.io~projected/kube-api-access-zb9jg:{mountpoint:/var/lib/kubelet/pods/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2/volumes/kubernetes.io~projected/kube-api-access-zb9jg major:0 minor:53 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6ae21aa1b206054e0d0d96d92dbeb4e3bd09457c5a179f8ddf1e78577e672da5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8ab2f1351a01534c9adc3608b674d8b322ae63b37c19a30879603d94fcedff04/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/51554f1eeb4c9bb705308aef85ceecf616f94bf431c9c70db72c7ef033bd9797/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/672decf9422e18f6ce45335e01ecc2a0a823be2f643aab0202d7495dd9dd07f6/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/d4433ba8e5bec767341ecd320abdcffc6ab8ef7ff8196541fc81c07da31dc0bc/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c7842d3e7315489945d9fa813964d570b28d3ba99ed1e8005be614011a204239/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/dfa8171195d1e348a87ce568e51d5767addeaf5e2265c8cf7b38b939dd7ceb03/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e30e69d63b1845a27b0a2bf8ebac995e50b1eb51f9801c56c73ce37222f5dcfe/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/f248ff39986a5381f7f8e31d8f3a9230144cd685a2b7e669db174f98faf69121/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/428fa6c2f5cf8a0ce40f0a7037cf5c8e45eb6b21376596c00e2d3b499a9f7466/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/aaecb635dfd04fcf624ad13e389625679381855066222741562d168570e39797/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/4a6f5afea79e301f5faf095e3c0933980e867ffeee6cf586a0ee6e96a5f11db5/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/88992f1317cb7ae5f597be958c9800d428fb6ebe11fbd12d8a5f07f49a9a22dc/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f2fbf30c0ec549bf9d46e5a77f82014469d4d209be8e1d0fe50f57ec5187b014/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3a6009ca8d688cdcb0285e4c528baff3f4867046c3200cc43703cc8ae7c53259/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1867ac6d768225d86e37606b4836794c907e2a3fdb7becbc28a075d7278b8042/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/cbec70d48453dea7c756cd074a221ba6c94e80cc821fd2037a75f43a468cffc7/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/c26b394d8cec42db185c69c66c8f0f8b2d444594e477a9ad04fbada9ae028522/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/5ab824a61d6d7ec1c048eb5e37071cf03cf13f56778fff1c08b7be317ad3db5d/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0a386aff10bba1af2a8bbbf281962fdd82e8fb54cad242a9b2ac05469e64c3b3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2ee4affce55dae547e94be6e2985fda317d57ed274d4d02c2e8371712ae88f16/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/bedb41fedee3561cf952a826b89ccf76f74e3215f2ce274c4d5697f0b75ad376/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/d744951b0ea623489201a6e0cf25e731e46009f42559505e9d23f71b93eca083/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/a999f150599a75d6d464cda1a80c5127c6a41fb8674bc25921c0a23d5873081a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/eb06a92b8bfa691c382341f9c463a696e5d7675b698dc413bb50f3de093d6cd8/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a7d5bd6670029cfc4dbe9bcfcef8410e4e064acc6ad4f3bae6991ac39896b72a/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/86ff81cf4e93ce31b9494faebbfdf1582be8c756a99579f847c15435d90cbfa0/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/434e57609ee77b9189f77a1aab1fcdf6ee74160a07053aeb3d7d6a871c560a16/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/6b42e111493a15b9820841293fd76da07053864335ff08fb0e5bc027ef2c65de/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/20b606dbe6c0466237b37f14aa4104c54b51103d59c5ec33a09bec08f03bbab3/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/e8abf2abfe5f40754926eb314d0f6598d32b92e213fb6cbd68342a8642251249/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/b928ad6ae4198108577cebe17ad45b88773cfbae20fadb61f9f0e45e2e815418/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/ad5f68af83d40168bfb1681059db9b981eef49c621682f3ab622a28ec389df92/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/2fd38f76f888a985f6192e9ce42a883889b3f8ef75cfc4d9a38cf33e57d020f5/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/61286cb35e2af612a754a24e88fc48a6c186c62920001d681a377b5d0620ff85/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/deb622ad8d3844e39bb0f062aa96d7c0caba1344c66fed1568eb94fb08fe4964/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/9002ee2c496cd69e01048ad8a974e8f5caef68e127b39b79d50dc7afad31ce23/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/6197aade4039493b57161761c852ae761d88295d6e02b75f4f84ac7ba4092134/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/fad22f205674a5d27f5effafe74db191bab169ac1c230136d0cb623d652b56e8/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/e8069464a5a4e71d7d0c8dda9bdc8d4933653f843bc558a449be5008127e3621/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/eb75fa616f05f3e60c5d2485cbc67767c96bbea8d54a120353832db3fb70560d/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/1957cd7712cc5939860f3b79ce7709f2001267ebf77c4f7fc60bc2a40a5fdab1/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/8b05cd551d398f5dd2b8d653da21e8a1b878d04271b84fd334ecf8584fc344ea/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/5c84d08c5b5f8e7b151300046235fb4f9c87369e46d6eedc5ae8e21d0e2309fc/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/d914d9dad97912b4cf48abe718934d289e881a9b8c40d7ac4f8da5ef6efddebe/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/ae1af19f7af65bf65ccdd92badede2d5b6216578bb6133c89e75c429af633dfa/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/0e90c1f482cd67a697eaea7f08fd9179f79d0157d92e17a284e745c7e60ac3fd/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/37a9dfb583ab10259e0f8be3d08e847c45708d1ff7118f9578579938d60257e9/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/1c34863faae876ffd989c2cefa5bbb834b430795c690956abb382d516e4cfecc/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/6ac1d2f77c6dd03a12e69083af229db9a72714499f715abaf3a8083cdb464644/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/81087b6cfc9c5a7a01c511cdbaf46a367dc0e84dc0533a963ec785dcde251fd9/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/623924b22f46b500e9136a20693b86ce25647de909c21b60f692f462a41ab592/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/30b66df9bca5044dcb8c64133250a13307d3701a87607b8035e7743832ef7a83/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/03958160cfc79a759d638bdea9e44f6f62215ae01f7c5fffd952b90ff4ad9ed4/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/59dd50d04fdc99dfbf98bd18697625bbde71c7c72e6fa2bddd2bd94a8b253bc8/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/ba204f8cfca79942b804483d92a4a1cbb35a9d6bef1e6fa9b34486748ec2ed53/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/12174ae0782476158a51b372e9cb705fdd33f36c09238f7ab8a35036d525c8d0/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/2e22c49aa709b4b85b60dc54ee7bc0148d4caf0c24bbfefa5a8cccfd673a6184/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/82ced9e261a891de0db7637d014e7d30368150df100dd3ae6764deb76fc274fc/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/fc2bda31e265191580c909748a1d894913575eccf5af0666fd5eb39c6462abce/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/d97d0ed277269b016c98621e02dca01025cc4f78a74b707b87bc2018cd05e896/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/b3702d168e0523a07f7c5535ed246c8ce555681129298083f605e94a2c97c282/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/6c2f2b3e3f4429f0ac5e19324011a531bfcfe77bb3db8f7c09e453fc455547bd/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/ea59c1aa5a17353996b8427aa7cf2667344da113a9916a1d62bfd68406592602/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/2b04635caa6ee91d872efbfc13e7440cb47ba75d6efd566b03476377b8783089/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/1133e77fa3b769ea75d9204a602868972ebe1a46331f88eef74da7fe617a1edd/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/049b3936161434c22eeaf631af0393539528024344729c8ae379984f2afd8896/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/d2a328a3415e3d2b44e0a8f413dac6dcaeae2ffd15693d757e2db2e656d71279/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/d01686784a5a5cdf7fe6c1c977618a5ca5c6e9d20f75c7204ab94eada6fa6ff7/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/87c62f54814c16bfdb2f6b24d2a770ceb8fad951177acb2fc249653fb009fe65/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/36e864f26471aaff356f2c83c5fae83700d3254232425275b997381a907e9e57/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/fb5033d01a4d61a49875405c263ad773323c0cdd91088753dfa7e2b0eea7217e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/4a8580ea78196ce27d21a602621a16e4fb2c8a58f3edbaad7b84c6b26a0e870e/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/d25117dc0483f461dd24836b244b9977da797622ca658068c39f98745387a8b4/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/29e2a6219c80cb2ef54a7891f9c0e444b4d3a1c95ca80c59d17967c33fce3fbb/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/fb0438fbab9c1d0d957b9a4404a78a490e018ec901501901bbb5d3dfa2e63d61/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/49d8dcafab40d9cee42d0b9ce69728c83dfcd27e7979b352452205d838831c18/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 01 12:22:38.555518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.554998    5725 manager.go:217] Machine: {Timestamp:2026-11-01 12:22:38.554514797 +0000 UTC m=+0.061706005 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101427 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9afed2e728359a1ef0545f81cc33 SystemUUID:ec2a9afe-d2e7-2835-9a1e-f0545f81cc33 BootID:29ce00db-89e8-478c-9b29-af788b1b0981 Filesystems:[{Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ec4e303cde0fe00ebe4cf59d0ff5ef5eaa4808adeda24c467573b2b44bbe584/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0215e6a2e22be96d6ed38a4f26830d0cf4fa8ec7763300c0e4b01dcdf876e710/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bad6344f-4c79-4140-abcc-2c4ae3d63049/volumes/kubernetes.io~projected/kube-api-access-vc6kk DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/533dfbadc7dd1c10cf42b78f0108ef1a2fb34aa2d1c1af95ef5fd5340cf3dbb5/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3888b182-8fb4-49bd-9af3-873bcc3efa92/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:409 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36/volumes/kubernetes.io~projected/kube-api-access-zwsm4 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b41f9fc-afce-464d-a18c-330694f80140/volumes/kubernetes.io~projected/kube-api-access-6rjsj DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c/volumes/kubernetes.io~projected/kube-api-access-h7csw DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39facba4-434d-476e-9e84-a8388d78e4e9/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47f53f35-b29e-4db5-91c8-0df05af362e7/volumes/kubernetes.io~projected/kube-api-access-x26dv DeviceMajor:0 DeviceMinor:280 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3888b182-8fb4-49bd-9af3-873bcc3efa92/volumes/kubernetes.io~projected/kube-api-access-cqbs7 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a024a7bf0ef503a9b7d0b9203af8167c404ea1e5cbcba3cd270ba730a79a3d14/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d176df9d439e171136a90a031eccdffbe8063a47d766bf548753b7e098f396fc/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ec05f2a91ebf5c7608b4a8a7ca18f3bbc6681d0d1214e8a46d3ac231a4cd096/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c00596c9-ffa3-461b-81be-7315388deaf1/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1efc2ea-a886-4c13-97a7-b1ddbd621feb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62e8634a9944589861c55c9744227e50c8ca411f4566a85ef8cd3802fb43bc0f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4cf38c23-496f-4661-85ce-586416959d95/volumes/kubernetes.io~projected/kube-api-access-jsjtg DeviceMajor:0 DeviceMinor:330 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf/volumes/kubernetes.io~projected/kube-api-access-6j6dv DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b43ebe-6a94-45f6-a613-0931f4607e9e/volumes/kubernetes.io~projected/kube-api-access-4t59b DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e5291295-5fd1-4e6e-b52b-c940c088168d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843/volumes/kubernetes.io~projected/kube-api-access-x2rjb DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03ef7e88-664f-413b-9424-c16a88d3b58c/volumes/kubernetes.io~projected/kube-api-access-hhrg8 DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1efc2ea-a886-4c13-97a7-b1ddbd621feb/volumes/kubernetes.io~projected/kube-api-access-m8lp6 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/690056c7-6829-4caf-b59e-e2650d20bb47/volumes/kubernetes.io~projected/kube-api-access-54rjm DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3953b512-91d0-401c-8397-0b9314e7a87b/volumes/kubernetes.io~projected/kube-api-access-rk8pw DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19e0e140-91e4-4d92-b1ac-0ea2115e3467/volumes/kubernetes.io~projected/kube-api-access-l4rtn DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/46433966-ea95-45b6-9092-c2f73bd7cd53/volumes/kubernetes.io~projected/kube-api-access-qlch9 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/693b0401-a9f9-41a4-b2e8-d3e3d6f37477/volumes/kubernetes.io~projected/kube-api-access-5gs79 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e90a2419d0c439d98c6bbe650e374cc26457eaf6b79fb89f8d1bf90dc041032f/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c09c0770-6cec-4827-9fb0-b07464653b27/volumes/kubernetes.io~projected/kube-api-access-fkkmg DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34c9bd1f-837b-4c0c-83da-3e5b09856707/volumes/kubernetes.io~projected/kube-api-access-6bkxh DeviceMajor:0 DeviceMinor:243 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3758a7d1-ccc2-4322-bd01-e742e5c11ff4/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45ce23c20cce1a88d17abd4055a31855e94aa59683f96cee242f45f86d17818f/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3d41b277973477bd26c72f0c47fbb5091275143533952735ede44b700b6ee37/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b1b9c0c9cc7dd8e7d04dd704ffc64bffc986a97bac42c4a6a8835321eb59e18/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/524fba8833ffac4a77f7472fa799f01603d6041c70eae7eaf6278c512c9c9dd1/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9d52d8311b3c108adedc1b192c47785e342ab5d1aeafd2d8eea6d734fcfb0ac/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8dd251ce1d9c2ca22b64a503d4545c365aa77e777504adfae1d628ce90e6b64/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fd19072fe78d3948dfeddfb760348f6ba8d08f7ac323b3839980b06cf5884f5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6cb3539-d27f-4954-9b4d-e730dfd5809d/volumes/kubernetes.io~projected/kube-api-access-qhf84 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7257c348c8d0768131c2427c447d061a1673df34fc8c14462bee77f3837dd452/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e7ee8e5a10fc26da1e61184d229a320b3115097df9f7acd06ea25b505b0ae62/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5cedf131-a296-4792-8328-f3ad030d3003/volumes/kubernetes.io~projected/kube-api-access-7bmhd DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36ea08297cbe07593735aaf7420137f4a1f2f8e85422bda88076bd722c990946/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c9208b0b436be94df86d2bb4a955ecb04f0ac2ad90da7afa1e23e96b3d1e459/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81fffd4e6988fb157febb30c1ca3844c26047f5cecb478327b88b6eb667108da/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3758a7d1-ccc2-4322-bd01-e742e5c11ff4/volumes/kubernetes.io~projected/kube-api-access-kzvmj DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8459c17c81a18ba53f4d923f2d486a46bfd55181679e028f9d18d4be6f0560d4/userdata/shm DeviceMajor:0 DeviceMinor:368 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76585ee8e6d28f78c7611868e223f47b5bd88670250b56909e45509680e609b3/userdata/shm DeviceMajor:0 DeviceMinor:410 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19e0e140-91e4-4d92-b1ac-0ea2115e3467/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2/volumes/kubernetes.io~projected/kube-api-access-zb9jg DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34c9bd1f-837b-4c0c-83da-3e5b09856707/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4ddc2b-35a6-4862-b64c-bb137efcdd69/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:367 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:384 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16839b5525d977e2ceb2cf35c97f94ddccca438d46194fac581c29d9c5798dab/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19e0e140-91e4-4d92-b1ac-0ea2115e3467/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/426e64a6-ecf0-471e-b130-15f85716445f/volumes/kubernetes.io~projected/kube-api-access-pqv4d DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/03ef7e88-664f-413b-9424-c16a88d3b58c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80e025ea2ea4cacbf74cce32958967710c1fd4d417b205cb290b96e28946efc1/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f6cfad4d2a416533c267b61ca1f75a9dbbf3e8efd36ad921a71a3591c5177cf/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28e2f6de099644659469607c349c0bd5570004e29c98a262e6f239abbf7e0a44/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbe766f5bec5c46aaa06e6ea37b0424e2f12dd74f5865c0d225e4e6df89050d8/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~projected/kube-api-access-9wr4q DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cef24fe79149cdd7d5c18280de7fe567075cd1a6ecec869b74f34b5f17fe665/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4ddc2b-35a6-4862-b64c-bb137efcdd69/volumes/kubernetes.io~projected/kube-api-access-4mxw6 DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8100acd04afd92612cb431b9ebf075b8d2d6d5de959dd4276468bf44fe4d78b0/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39facba4-434d-476e-9e84-a8388d78e4e9/volumes/kubernetes.io~projected/kube-api-access-4s67j DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a18ed8bb12322dbc7c4ee48db29d33cf28a4e5956e5f20f4908ddea6af352e3/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c33d8154e2667ba4e26fcab11d1d2fb2034477f3f6d80b23e29b2a4e20b7ee2/userdata/shm DeviceMajor:0 DeviceMinor:378 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b41f9fc-afce-464d-a18c-330694f80140/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0e7ee8e5a10fc26 MacAddress:32:cd:85:96:55:30 Speed:10000 Mtu:8901} {Name:28e2f6de0996446 MacAddress:aa:fc:7c:9a:4e:cb Speed:10000 Mtu:8901} {Name:2a18ed8bb12322d MacAddress:76:88:ce:5a:8e:d7 Speed:10000 Mtu:8901} {Name:45ce23c20cce1a8 MacAddress:c6:98:8b:71:79:7d Speed:10000 Mtu:8901} {Name:4cef24fe79149cd MacAddress:c2:e2:e6:b0:1b:5c Speed:10000 Mtu:8901} {Name:524fba8833ffac4 MacAddress:2a:19:b3:cc:ff:c5 Speed:10000 Mtu:8901} {Name:533dfbadc7dd1c1 MacAddress:e2:f0:64:d1:3b:53 Speed:10000 Mtu:8901} {Name:5f6cfad4d2a4165 MacAddress:9a:34:88:3c:42:51 Speed:10000 Mtu:8901} {Name:62e8634a9944589 MacAddress:3a:39:4b:73:b3:20 Speed:10000 Mtu:8901} {Name:6ec05f2a91ebf5c MacAddress:0a:3a:60:f5:13:a5 Speed:10000 Mtu:8901} {Name:7257c348c8d0768 MacAddress:26:22:66:a3:ae:93 Speed:10000 Mtu:8901} {Name:76585ee8e6d28f7 MacAddress:4e:90:15:91:15:f8 Speed:10000 Mtu:8901} {Name:7b1b9c0c9cc7dd8 MacAddress:aa:e2:8f:bb:06:84 Speed:10000 Mtu:8901} {Name:8100acd04afd926 MacAddress:1a:98:b7:66:7b:3b Speed:10000 Mtu:8901} {Name:8459c17c81a18ba MacAddress:4e:d3:2e:af:f5:10 Speed:10000 Mtu:8901} {Name:8c33d8154e2667b MacAddress:a2:fb:ec:e4:c4:0f Speed:10000 Mtu:8901} {Name:b8dd251ce1d9c2c MacAddress:2e:00:93:1a:17:81 Speed:10000 Mtu:8901} {Name:bbe766f5bec5c46 MacAddress:e6:86:88:7a:50:fa Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d3:c3:72:8d Speed:0 Mtu:9001} {Name:br-int MacAddress:66:5a:f8:12:1d:61 Speed:0 Mtu:8901} {Name:e90a2419d0c439d MacAddress:f2:c7:e2:3e:b9:c5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d3:c3:72:8d Speed:0 Mtu:9001} {Name:f3d41b277973477 MacAddress:26:1a:cb:d5:9d:49 Speed:10000 Mtu:8901} {Name:f9d52d8311b3c10 MacAddress:42:93:ac:e2:aa:7c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:12:90:9b:6b:94:98 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:b2:db:c9:fa:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 12:22:38.555749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.555523    5725 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 12:22:38.555749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.555573    5725 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 12:22:38.555941 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.555909    5725 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 12:22:38.556092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.555942    5725 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-76.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 12:22:38.556134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556095    5725 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 12:22:38.556134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556103    5725 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 12:22:38.556134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556114    5725 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 12:22:38.556134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556127    5725 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 12:22:38.556239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556148    5725 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 12:22:38.556319 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556310    5725 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 12:22:38.556433 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556424    5725 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 12:22:38.556469 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556460    5725 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 12:22:38.556500 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556479    5725 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 12:22:38.556527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556500    5725 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 12:22:38.556527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556512    5725 kubelet.go:405] "Adding apiserver pod source"
Nov 01 12:22:38.556527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.556520    5725 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 12:22:38.558026 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.558009    5725 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 12:22:38.558091 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.558028    5725 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 12:22:38.558598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.558579    5725 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 12:22:38.562085 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562033    5725 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 12:22:38.562085 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562075    5725 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 12:22:38.562311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562296    5725 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 12:22:38.562686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562669    5725 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 12:22:38.562764 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562710    5725 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 12:22:38.562846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562835    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 12:22:38.562943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562854    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 12:22:38.562943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562877    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 12:22:38.562943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562886    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 12:22:38.562943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562895    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 12:22:38.562943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562903    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 12:22:38.562943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562913    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 12:22:38.562943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562921    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 12:22:38.562943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562932    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 12:22:38.562943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562941    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 12:22:38.563356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562955    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 12:22:38.563356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.562969    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 12:22:38.563356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.563007    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 12:22:38.563356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.563016    5725 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 12:22:38.566736 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.566719    5725 server.go:1263] "Started kubelet"
Nov 01 12:22:38.566846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.566810    5725 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 12:22:38.567222 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.567177    5725 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 12:22:38.567304 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.567241    5725 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 12:22:38.567298 ip-10-0-138-76 systemd[1]: Started Kubernetes Kubelet.
Nov 01 12:22:38.567540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.567518    5725 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 12:22:38.570724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.570707    5725 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 12:22:38.570856 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.570841    5725 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 12:22:38.571989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.571517    5725 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 12:22:38.571989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.571559    5725 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 12:16:27 +0000 UTC" deadline="2028-04-01 22:32:50.060335099 +0000 UTC"
Nov 01 12:22:38.571989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.571593    5725 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12418h10m11.488744524s"
Nov 01 12:22:38.571989 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:38.571627    5725 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-76.us-west-2.compute.internal\" not found"
Nov 01 12:22:38.571989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.571663    5725 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 12:22:38.571989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.571673    5725 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 12:22:38.571989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.571874    5725 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 12:22:38.574991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.574973    5725 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 12:22:38.574991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.574989    5725 factory.go:55] Registering systemd factory
Nov 01 12:22:38.574991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.574996    5725 factory.go:223] Registration of the systemd container factory successfully
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575089    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34c9bd1f-837b-4c0c-83da-3e5b09856707" volumeName="kubernetes.io/secret/34c9bd1f-837b-4c0c-83da-3e5b09856707-cert" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575117    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426e64a6-ecf0-471e-b130-15f85716445f" volumeName="kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-stats-auth" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575127    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" volumeName="kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575136    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1efc2ea-a886-4c13-97a7-b1ddbd621feb" volumeName="kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-service-ca-bundle" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575144    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e0e140-91e4-4d92-b1ac-0ea2115e3467" volumeName="kubernetes.io/secret/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575151    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34c9bd1f-837b-4c0c-83da-3e5b09856707" volumeName="kubernetes.io/projected/34c9bd1f-837b-4c0c-83da-3e5b09856707-kube-api-access-6bkxh" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575158    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46433966-ea95-45b6-9092-c2f73bd7cd53" volumeName="kubernetes.io/projected/46433966-ea95-45b6-9092-c2f73bd7cd53-kube-api-access-qlch9" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575165    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" volumeName="kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575174    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6cb3539-d27f-4954-9b4d-e730dfd5809d" volumeName="kubernetes.io/projected/b6cb3539-d27f-4954-9b4d-e730dfd5809d-kube-api-access-qhf84" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575181    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffad4d99-42c6-4151-a4cf-e22b3ecd03e2" volumeName="kubernetes.io/configmap/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-iptables-alerter-script" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575187    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ef7e88-664f-413b-9424-c16a88d3b58c" volumeName="kubernetes.io/empty-dir/03ef7e88-664f-413b-9424-c16a88d3b58c-data-volume" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575194    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b41f9fc-afce-464d-a18c-330694f80140" volumeName="kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-trusted-ca" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575201    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b41f9fc-afce-464d-a18c-330694f80140" volumeName="kubernetes.io/projected/9b41f9fc-afce-464d-a18c-330694f80140-kube-api-access-6rjsj" seLinuxMountContext=""
Nov 01 12:22:38.575195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575203    5725 factory.go:153] Registering CRI-O factory
Nov 01 12:22:38.575635 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575210    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4ddc2b-35a6-4862-b64c-bb137efcdd69" volumeName="kubernetes.io/empty-dir/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-tmp-dir" seLinuxMountContext=""
Nov 01 12:22:38.575635 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575220    5725 factory.go:223] Registration of the crio container factory successfully
Nov 01 12:22:38.575635 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575229    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b43ebe-6a94-45f6-a613-0931f4607e9e" volumeName="kubernetes.io/configmap/f8b43ebe-6a94-45f6-a613-0931f4607e9e-serviceca" seLinuxMountContext=""
Nov 01 12:22:38.575635 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575237    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3888b182-8fb4-49bd-9af3-873bcc3efa92" volumeName="kubernetes.io/projected/3888b182-8fb4-49bd-9af3-873bcc3efa92-kube-api-access-cqbs7" seLinuxMountContext=""
Nov 01 12:22:38.575635 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575245    5725 factory.go:103] Registering Raw factory
Nov 01 12:22:38.575635 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575261    5725 manager.go:1199] Started watching for new ooms in manager
Nov 01 12:22:38.576099 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.575244    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426e64a6-ecf0-471e-b130-15f85716445f" volumeName="kubernetes.io/projected/426e64a6-ecf0-471e-b130-15f85716445f-kube-api-access-pqv4d" seLinuxMountContext=""
Nov 01 12:22:38.576159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576106    5725 manager.go:319] Starting recovery of all containers
Nov 01 12:22:38.576159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576108    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f53f35-b29e-4db5-91c8-0df05af362e7" volumeName="kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv" seLinuxMountContext=""
Nov 01 12:22:38.576159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576126    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cedf131-a296-4792-8328-f3ad030d3003" volumeName="kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-multus-daemon-config" seLinuxMountContext=""
Nov 01 12:22:38.576159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576138    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cedf131-a296-4792-8328-f3ad030d3003" volumeName="kubernetes.io/projected/5cedf131-a296-4792-8328-f3ad030d3003-kube-api-access-7bmhd" seLinuxMountContext=""
Nov 01 12:22:38.576159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576150    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b43ebe-6a94-45f6-a613-0931f4607e9e" volumeName="kubernetes.io/projected/f8b43ebe-6a94-45f6-a613-0931f4607e9e-kube-api-access-4t59b" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576162    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3888b182-8fb4-49bd-9af3-873bcc3efa92" volumeName="kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576173    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39facba4-434d-476e-9e84-a8388d78e4e9" volumeName="kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576186    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cedf131-a296-4792-8328-f3ad030d3003" volumeName="kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-cni-binary-copy" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576197    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" volumeName="kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576208    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" volumeName="kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576220    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bad6344f-4c79-4140-abcc-2c4ae3d63049" volumeName="kubernetes.io/projected/bad6344f-4c79-4140-abcc-2c4ae3d63049-kube-api-access-vc6kk" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576232    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ef7e88-664f-413b-9424-c16a88d3b58c" volumeName="kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576248    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843" volumeName="kubernetes.io/projected/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-kube-api-access-x2rjb" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576260    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843" volumeName="kubernetes.io/secret/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-serving-cert" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576270    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426e64a6-ecf0-471e-b130-15f85716445f" volumeName="kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-default-certificate" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576280    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" volumeName="kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576293    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1efc2ea-a886-4c13-97a7-b1ddbd621feb" volumeName="kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576304    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1efc2ea-a886-4c13-97a7-b1ddbd621feb" volumeName="kubernetes.io/secret/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-serving-cert" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576315    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3758a7d1-ccc2-4322-bd01-e742e5c11ff4" volumeName="kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576326    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39facba4-434d-476e-9e84-a8388d78e4e9" volumeName="kubernetes.io/configmap/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-config" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576336    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36" volumeName="kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-tmp" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576348    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36" volumeName="kubernetes.io/projected/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-kube-api-access-zwsm4" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576360    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" volumeName="kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576372    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" volumeName="kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets" seLinuxMountContext=""
Nov 01 12:22:38.576398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576408    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="370a3bfd-1661-4e44-b5d5-5f6b83db1f6c" volumeName="kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576420    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="690056c7-6829-4caf-b59e-e2650d20bb47" volumeName="kubernetes.io/projected/690056c7-6829-4caf-b59e-e2650d20bb47-kube-api-access-54rjm" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576431    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39facba4-434d-476e-9e84-a8388d78e4e9" volumeName="kubernetes.io/projected/39facba4-434d-476e-9e84-a8388d78e4e9-kube-api-access-4s67j" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576443    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c" volumeName="kubernetes.io/configmap/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-config" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576454    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c" volumeName="kubernetes.io/secret/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-serving-cert" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576465    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00596c9-ffa3-461b-81be-7315388deaf1" volumeName="kubernetes.io/configmap/c00596c9-ffa3-461b-81be-7315388deaf1-nginx-conf" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576476    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5291295-5fd1-4e6e-b52b-c940c088168d" volumeName="kubernetes.io/configmap/e5291295-5fd1-4e6e-b52b-c940c088168d-konnectivity-ca" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576488    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e0e140-91e4-4d92-b1ac-0ea2115e3467" volumeName="kubernetes.io/projected/19e0e140-91e4-4d92-b1ac-0ea2115e3467-kube-api-access-l4rtn" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576499    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426e64a6-ecf0-471e-b130-15f85716445f" volumeName="kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576512    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b41f9fc-afce-464d-a18c-330694f80140" volumeName="kubernetes.io/secret/9b41f9fc-afce-464d-a18c-330694f80140-serving-cert" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576524    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5291295-5fd1-4e6e-b52b-c940c088168d" volumeName="kubernetes.io/secret/e5291295-5fd1-4e6e-b52b-c940c088168d-agent-certs" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576536    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4ddc2b-35a6-4862-b64c-bb137efcdd69" volumeName="kubernetes.io/configmap/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-config-volume" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576549    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4ddc2b-35a6-4862-b64c-bb137efcdd69" volumeName="kubernetes.io/projected/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-kube-api-access-4mxw6" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576563    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ef7e88-664f-413b-9424-c16a88d3b58c" volumeName="kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576578    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf" volumeName="kubernetes.io/projected/3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf-kube-api-access-6j6dv" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576590    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" volumeName="kubernetes.io/projected/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-kube-api-access-5gs79" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576612    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36" volumeName="kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-tuned" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576624    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" volumeName="kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576637    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1efc2ea-a886-4c13-97a7-b1ddbd621feb" volumeName="kubernetes.io/projected/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-kube-api-access-m8lp6" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576652    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4ddc2b-35a6-4862-b64c-bb137efcdd69" volumeName="kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576665    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b41f9fc-afce-464d-a18c-330694f80140" volumeName="kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-config" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576677    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1efc2ea-a886-4c13-97a7-b1ddbd621feb" volumeName="kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-tmp" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576691    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffad4d99-42c6-4151-a4cf-e22b3ecd03e2" volumeName="kubernetes.io/projected/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-kube-api-access-zb9jg" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576705    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ef7e88-664f-413b-9424-c16a88d3b58c" volumeName="kubernetes.io/projected/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-api-access-hhrg8" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576717    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426e64a6-ecf0-471e-b130-15f85716445f" volumeName="kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576732    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3758a7d1-ccc2-4322-bd01-e742e5c11ff4" volumeName="kubernetes.io/configmap/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-telemetry-config" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576745    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3758a7d1-ccc2-4322-bd01-e742e5c11ff4" volumeName="kubernetes.io/projected/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-kube-api-access-kzvmj" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576757    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e0e140-91e4-4d92-b1ac-0ea2115e3467" volumeName="kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-env-overrides" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576770    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e0e140-91e4-4d92-b1ac-0ea2115e3467" volumeName="kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-config" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576783    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3953b512-91d0-401c-8397-0b9314e7a87b" volumeName="kubernetes.io/projected/3953b512-91d0-401c-8397-0b9314e7a87b-kube-api-access-rk8pw" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576794    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843" volumeName="kubernetes.io/configmap/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-config" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576809    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" volumeName="kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576822    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cf38c23-496f-4661-85ce-586416959d95" volumeName="kubernetes.io/projected/4cf38c23-496f-4661-85ce-586416959d95-kube-api-access-jsjtg" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576834    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" volumeName="kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-binary-copy" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576846    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" volumeName="kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576858    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c00596c9-ffa3-461b-81be-7315388deaf1" volumeName="kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576870    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c09c0770-6cec-4827-9fb0-b07464653b27" volumeName="kubernetes.io/empty-dir/c09c0770-6cec-4827-9fb0-b07464653b27-tmp-dir" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576881    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1efc2ea-a886-4c13-97a7-b1ddbd621feb" volumeName="kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-snapshots" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576893    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e0e140-91e4-4d92-b1ac-0ea2115e3467" volumeName="kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576904    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c" volumeName="kubernetes.io/projected/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-kube-api-access-h7csw" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576915    5725 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c09c0770-6cec-4827-9fb0-b07464653b27" volumeName="kubernetes.io/projected/c09c0770-6cec-4827-9fb0-b07464653b27-kube-api-access-fkkmg" seLinuxMountContext=""
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576928    5725 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 12:22:38.577349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.576935    5725 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 12:22:38.596930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.596901    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:22:38.598175 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.598147    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:22:38.606011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.605972    5725 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 12:22:38.608391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.608366    5725 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 12:22:38.608486 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.608396    5725 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 12:22:38.608486 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.608418    5725 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 12:22:38.608486 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:38.608459    5725 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 12:22:38.632999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.632961    5725 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="1f6980da3ee79dc1ef05a975c87c4c3a1dfd5135a9e9c20d01ef8fd9af0b848b" exitCode=0
Nov 01 12:22:38.632999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.633000    5725 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="ebfc6227506f940fd92df54b92a00ddd407817bd6864c4523ac15e90c9f86ebb" exitCode=0
Nov 01 12:22:38.633192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.633012    5725 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="bdc2237f4c9d898370dd53fe23e40e9274b9605464b95f590bb71e5c48d0fa39" exitCode=0
Nov 01 12:22:38.633192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.633024    5725 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="9c2941adbe17205cac228f982c46cf06d7b0cf951cfc2ea56d8bf4cea21f60b7" exitCode=0
Nov 01 12:22:38.633192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.633037    5725 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="ff4dc214db264f263966e2a7e917c20bcc86342ce1372f41c2b590ddef3499d9" exitCode=0
Nov 01 12:22:38.633192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.633077    5725 generic.go:356] "Generic (PLEG): container finished" podID="693b0401-a9f9-41a4-b2e8-d3e3d6f37477" containerID="789ff9be36678b829ce7c675364e2835643879de056bbdb7e84446d6b786790e" exitCode=0
Nov 01 12:22:38.636523 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.636501    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:22:38.636628 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.636530    5725 generic.go:356] "Generic (PLEG): container finished" podID="9b41f9fc-afce-464d-a18c-330694f80140" containerID="df2d582fa3b0ee2aa06cd65a280dac02c8b043b6d7a355aef887ebb44c671f0c" exitCode=255
Nov 01 12:22:38.645637 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.645614    5725 generic.go:356] "Generic (PLEG): container finished" podID="839bae4638948a932829e431666671db" containerID="a3fa00c99728dfe064e0bca25339ac6d62572dc59a4676bc3f5fa2f1bb94fddf" exitCode=0
Nov 01 12:22:38.671719 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:38.671691    5725 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-76.us-west-2.compute.internal\" not found"
Nov 01 12:22:38.687280 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.687255    5725 manager.go:324] Recovery completed
Nov 01 12:22:38.707199 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.707172    5725 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:38.708531 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:38.708513    5725 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 12:22:38.709400 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.709377    5725 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:38.709484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.709416    5725 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:38.709484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.709433    5725 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:38.713179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713153    5725 cpu_manager.go:226] "Starting" policy="none"
Nov 01 12:22:38.713179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713181    5725 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 12:22:38.713333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713205    5725 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 12:22:38.713423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713403    5725 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 12:22:38.713482 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713424    5725 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 12:22:38.713482 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713450    5725 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 12:22:38.713482 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713458    5725 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 12:22:38.713482 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713466    5725 policy_none.go:50] "Start"
Nov 01 12:22:38.713645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713487    5725 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 12:22:38.713645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.713501    5725 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 12:22:38.714093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.714069    5725 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:38.715254 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.715235    5725 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:38.715328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.715273    5725 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:38.715328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.715287    5725 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:38.726881 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.726855    5725 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 12:22:38.726881 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.726882    5725 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 12:22:38.727079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.726893    5725 policy_none.go:44] "Start"
Nov 01 12:22:38.732214 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.732182    5725 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 12:22:38.735445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.735430    5725 manager.go:341] "Starting Device Plugin manager"
Nov 01 12:22:38.735523 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:38.735468    5725 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 12:22:38.735523 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.735482    5725 server.go:87] "Starting device plugin registration server"
Nov 01 12:22:38.735757 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.735744    5725 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 12:22:38.735803 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.735757    5725 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 12:22:38.735871 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.735853    5725 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 12:22:38.736010 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.735970    5725 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 12:22:38.736095 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.736027    5725 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 12:22:38.736095 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.736035    5725 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 12:22:38.736194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.736113    5725 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T12:22:38.736008542Z","UUID":"58085b65-c2de-450d-9f02-5dc39ea97463","Handler":null,"Name":"","Endpoint":""}
Nov 01 12:22:38.737488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.737467    5725 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 12:22:38.737592 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.737511    5725 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 12:22:38.737946 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:38.737925    5725 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 12:22:38.738161 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:38.737967    5725 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-76.us-west-2.compute.internal\" not found"
Nov 01 12:22:38.758412 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.758384    5725 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:22:38.758927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.758911    5725 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 12:22:38.758927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.758914    5725 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:22:38.803982 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.803918    5725 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:22:38.810032 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.810011    5725 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:22:38.836167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.836147    5725 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:38.837240 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.837222    5725 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:38.837316 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.837254    5725 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:38.837316 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.837264    5725 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-76.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:38.837316 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.837310    5725 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:38.909634 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.909581    5725 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"]
Nov 01 12:22:38.910273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.910224    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"d08d3cfd44df29731faafa69be6f8df6","Type":"ContainerStarted","Data":"c35c3ab8e4c7d4c441000795553d44e1b11b9eeee4d8dc323ef6dcb19e1d14f3"}
Nov 01 12:22:38.910336 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.910286    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"d08d3cfd44df29731faafa69be6f8df6","Type":"ContainerStarted","Data":"36ea08297cbe07593735aaf7420137f4a1f2f8e85422bda88076bd722c990946"}
Nov 01 12:22:38.910369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.910289    5725 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:38.910406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.910376    5725 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:38.910406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.910390    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"839bae4638948a932829e431666671db","Type":"ContainerStarted","Data":"1a925b09a5c452a9f252159e905fff6ce02448a09887f367581b4d1d39267406"}
Nov 01 12:22:38.910490 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.910412    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"839bae4638948a932829e431666671db","Type":"ContainerDied","Data":"a3fa00c99728dfe064e0bca25339ac6d62572dc59a4676bc3f5fa2f1bb94fddf"}
Nov 01 12:22:38.910490 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.910432    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal" event={"ID":"839bae4638948a932829e431666671db","Type":"ContainerStarted","Data":"a024a7bf0ef503a9b7d0b9203af8167c404ea1e5cbcba3cd270ba730a79a3d14"}
Nov 01 12:22:38.940153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.940128    5725 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:38.940229 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.940206    5725 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:38.967422 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.967396    5725 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:22:38.967549 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:38.967445    5725 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:38.968084 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:38.968043    5725 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:22:38.968119 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:38.968111    5725 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.060981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.060881    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d08d3cfd44df29731faafa69be6f8df6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"d08d3cfd44df29731faafa69be6f8df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.060981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.060932    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.060981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.060953    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.162153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.162110    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.162153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.162155    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.162353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.162181    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d08d3cfd44df29731faafa69be6f8df6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"d08d3cfd44df29731faafa69be6f8df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.162353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.162231    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d08d3cfd44df29731faafa69be6f8df6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"d08d3cfd44df29731faafa69be6f8df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.162353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.162266    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.162353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.162233    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/839bae4638948a932829e431666671db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" (UID: \"839bae4638948a932829e431666671db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.557613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.557575    5725 apiserver.go:52] "Watching apiserver"
Nov 01 12:22:39.676636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.676605    5725 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.676795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.676659    5725 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.703179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.703155    5725 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 12:22:39.703799 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.703776    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-vrjcf","openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn","kube-system/kas-connection-checker-685895c7c9-fg5h6","openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p","openshift-multus/multus-lxjxw","openshift-network-diagnostics/network-check-target-6zjj4","kube-system/kas-connection-checker-685895c7c9-rsltv","openshift-multus/multus-additional-cni-plugins-tg2vq","openshift-network-operator/iptables-alerter-xwv48","openshift-cluster-node-tuning-operator/tuned-dvz6d","kube-system/global-pull-secret-syncer-v7lhr","kube-system/konnectivity-agent-bm5xm","openshift-multus/network-metrics-daemon-p779s","openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml","openshift-dns/dns-default-xm57l","openshift-ingress/router-default-79dd7d46d9-xfhjz","kube-system/kas-connection-checker-685895c7c9-tsmbq","openshift-image-registry/image-registry-84fbdf67f8-6fmbg","openshift-console-operator/console-operator-7cd56b8dbc-slc7h","openshift-insights/insights-runtime-extractor-9xgxt","openshift-image-registry/node-ca-fpq52","kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal","openshift-network-console/networking-console-plugin-86c478bb88-744lb","openshift-insights/insights-operator-7c4986d449-rdmlg","openshift-dns/node-resolver-kfdrq","openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd","openshift-ingress-canary/ingress-canary-x6gnr"]
Nov 01 12:22:39.733796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.733770    5725 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:22:39.733930 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:39.733821    5725 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.734275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.734262    5725 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:22:39.734340 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:22:39.734296    5725 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-76.us-west-2.compute.internal"
Nov 01 12:22:39.754578 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.754557    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.754646 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.754593    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.754857 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.754840    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.755189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.755169    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.755336 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.755312    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.755445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.755349    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.755445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.755380    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 12:22:39.755445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.755381    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 12:22:39.755445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.755394    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 12:22:39.755598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.755577    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.755727 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.755714    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 12:22:39.755795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.755754    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.756202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.756183    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 12:22:39.756309 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.756264    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-d6zhm\""
Nov 01 12:22:39.756387 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.756371    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 12:22:39.774312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.774290    5725 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 12:22:39.780699 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.780681    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 12:22:39.780796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.780751    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 12:22:39.780840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.780792    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 12:22:39.800164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.800136    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 12:22:39.800164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.800145    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jtrgb\""
Nov 01 12:22:39.800277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.800145    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.800989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.800976    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 12:22:39.801030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.801005    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wkt5z\""
Nov 01 12:22:39.801543 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.801531    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.801583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.801566    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.801700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.801687    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8q978\""
Nov 01 12:22:39.801819 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.801807    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 12:22:39.801819 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.801814    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.801997 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.801984    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.801997 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.801990    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-v2t2t\""
Nov 01 12:22:39.802132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802012    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.802132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802016    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 12:22:39.802132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802071    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-62ffm\""
Nov 01 12:22:39.802132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802082    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jxlnp\""
Nov 01 12:22:39.802318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802217    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zhblh\""
Nov 01 12:22:39.802318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802225    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2z6g\""
Nov 01 12:22:39.802318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802277    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 12:22:39.802318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802314    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.802505 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802317    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 12:22:39.802505 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802328    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.802505 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802449    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.802655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802542    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 12:22:39.802655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802593    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 12:22:39.802655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802612    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 12:22:39.802655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802611    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9cl4q\""
Nov 01 12:22:39.802655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802616    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.802655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802624    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802695    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802732    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802733    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802748    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802757    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802779    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802804    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802884    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802886    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.802943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802931    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 12:22:39.803406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802970    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.803406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802979    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.803406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.802987    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 12:22:39.803406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803015    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 12:22:39.803406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803044    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.803406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803261    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sfjvd\""
Nov 01 12:22:39.803406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803326    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 12:22:39.803686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803496    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 12:22:39.803686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803629    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.803686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803650    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qrhfv\""
Nov 01 12:22:39.803781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803692    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-c2rwl\""
Nov 01 12:22:39.803781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803700    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 12:22:39.803781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803732    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7twmh\""
Nov 01 12:22:39.803918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.803905    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 12:22:39.849016 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.848937    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.849016 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.848938    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 12:22:39.849210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849043    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.849210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849080    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 12:22:39.849210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849190    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.849210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849206    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 12:22:39.849391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849217    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.849391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849228    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 12:22:39.849391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849356    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 12:22:39.849526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849407    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.849526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849466    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z9pcl\""
Nov 01 12:22:39.849526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849499    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhf7p\""
Nov 01 12:22:39.849654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849525    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzkjs\""
Nov 01 12:22:39.849654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849536    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.849654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849555    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nsq77\""
Nov 01 12:22:39.849654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849572    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9t2fc\""
Nov 01 12:22:39.849654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849596    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 12:22:39.849654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849604    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 12:22:39.849654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849632    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 12:22:39.849654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849653    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lt5kb\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849656    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-m68cz\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849669    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kqn6l\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849692    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849712    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-phch9\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849773    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fj2dt\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849815    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849853    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849860    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849855    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849886    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-svx22\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849905    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849982    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.850001    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 12:22:39.850001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.849986    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 12:22:39.850672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.850022    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 12:22:39.850672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.850081    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-86gmg\""
Nov 01 12:22:39.850672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.850147    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-thbc5\""
Nov 01 12:22:39.850672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.850563    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 12:22:39.865512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.865483    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6dv\" (UniqueName: \"kubernetes.io/projected/3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf-kube-api-access-6j6dv\") pod \"kas-connection-checker-685895c7c9-rsltv\" (UID: \"3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf\") " pod="kube-system/kas-connection-checker-685895c7c9-rsltv"
Nov 01 12:22:39.865604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.865524    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-k8s-cni-cncf-io\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.865604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.865544    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-etc-kubernetes\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.865604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.865566    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvmj\" (UniqueName: \"kubernetes.io/projected/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-kube-api-access-kzvmj\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:39.865712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.865626    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c09c0770-6cec-4827-9fb0-b07464653b27-hosts-file\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:22:39.865763 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.865736    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-hostroot\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.865837 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.865823    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-multus-certs\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.865887 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.865865    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9jg\" (UniqueName: \"kubernetes.io/projected/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-kube-api-access-zb9jg\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:22:39.865935 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.865895    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03ef7e88-664f-413b-9424-c16a88d3b58c-crio-socket\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.866032 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866009    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8lp6\" (UniqueName: \"kubernetes.io/projected/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-kube-api-access-m8lp6\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.866099 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866070    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-lib-modules\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.866132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866104    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-kubelet\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.866160 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866131    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.866190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866158    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjsj\" (UniqueName: \"kubernetes.io/projected/9b41f9fc-afce-464d-a18c-330694f80140-kube-api-access-6rjsj\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:39.866237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866187    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkmg\" (UniqueName: \"kubernetes.io/projected/c09c0770-6cec-4827-9fb0-b07464653b27-kube-api-access-fkkmg\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:22:39.866237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866212    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-registration-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.866315 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866238    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-serving-cert\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.866315 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866298    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8b43ebe-6a94-45f6-a613-0931f4607e9e-host\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:22:39.866966 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866936    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t59b\" (UniqueName: \"kubernetes.io/projected/f8b43ebe-6a94-45f6-a613-0931f4607e9e-kube-api-access-4t59b\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:22:39.867094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.866973    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-log-socket\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.867094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867012    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.867094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867044    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-conf\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.867244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867101    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.867244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867140    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-etc-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.867244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867166    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-tmp\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.867244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867195    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b41f9fc-afce-464d-a18c-330694f80140-serving-cert\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:39.867244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867220    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-serving-cert\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867219    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-tuned\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867258    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-tmp\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867284    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-tuned\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867288    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867308    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867327    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867346    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s67j\" (UniqueName: \"kubernetes.io/projected/39facba4-434d-476e-9e84-a8388d78e4e9-kube-api-access-4s67j\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867381    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-snapshots\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867433    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-run\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.867478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867468    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7csw\" (UniqueName: \"kubernetes.io/projected/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-kube-api-access-h7csw\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867486    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867494    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rjm\" (UniqueName: \"kubernetes.io/projected/690056c7-6829-4caf-b59e-e2650d20bb47-kube-api-access-54rjm\") pod \"volume-data-source-validator-58c9f5f46d-j4vzd\" (UID: \"690056c7-6829-4caf-b59e-e2650d20bb47\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867522    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867537    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867541    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-ovn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867566    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-node-log\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867583    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-config-volume\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867604    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867624    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-var-lib-kubelet\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867641    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867722    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-snapshots\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867730    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b41f9fc-afce-464d-a18c-330694f80140-serving-cert\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867782    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-env-overrides\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867786    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867803    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-original-pull-secret\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867823    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c00596c9-ffa3-461b-81be-7315388deaf1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867866    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wr4q\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867897    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-bin\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867917    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovn-node-metrics-cert\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867934    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867948    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-tmp-dir\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867966    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-serving-cert\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867981    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867995    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwsm4\" (UniqueName: \"kubernetes.io/projected/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-kube-api-access-zwsm4\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868010    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-systemd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868034    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-config-volume\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:39.868066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868037    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868183    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-metrics-tls\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868233    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovn-node-metrics-cert\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868275    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjtg\" (UniqueName: \"kubernetes.io/projected/4cf38c23-496f-4661-85ce-586416959d95-kube-api-access-jsjtg\") pod \"migrator-66df5d496b-twpxk\" (UID: \"4cf38c23-496f-4661-85ce-586416959d95\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868305    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-modprobe-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.867991    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-env-overrides\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868301    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03ef7e88-664f-413b-9424-c16a88d3b58c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868358    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrg8\" (UniqueName: \"kubernetes.io/projected/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-api-access-hhrg8\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868403    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-serving-cert\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868633    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-tmp-dir\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:39.869267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868858    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c00596c9-ffa3-461b-81be-7315388deaf1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.869722    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.868737    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.869887    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.869939    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.869972    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5291295-5fd1-4e6e-b52b-c940c088168d-konnectivity-ca\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870021    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870086    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870118    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-config\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870169    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmhd\" (UniqueName: \"kubernetes.io/projected/5cedf131-a296-4792-8328-f3ad030d3003-kube-api-access-7bmhd\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870201    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-sys-fs\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870249    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8b43ebe-6a94-45f6-a613-0931f4607e9e-serviceca\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870279    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-cni-binary-copy\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870320    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870350    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34c9bd1f-837b-4c0c-83da-3e5b09856707-cert\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870396    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-netns\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870428    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-multus\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870477    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5291295-5fd1-4e6e-b52b-c940c088168d-agent-certs\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870504    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-stats-auth\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870548    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-sys\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870581    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870630    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-trusted-ca\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870662    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhf84\" (UniqueName: \"kubernetes.io/projected/b6cb3539-d27f-4954-9b4d-e730dfd5809d-kube-api-access-qhf84\") pod \"kas-connection-checker-685895c7c9-fg5h6\" (UID: \"b6cb3539-d27f-4954-9b4d-e730dfd5809d\") " pod="kube-system/kas-connection-checker-685895c7c9-fg5h6"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870698    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-multus-daemon-config\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870730    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-device-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870757    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-system-cni-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870785    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-host-slash\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870834    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870866    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqv4d\" (UniqueName: \"kubernetes.io/projected/426e64a6-ecf0-471e-b130-15f85716445f-kube-api-access-pqv4d\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870907    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870939    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cnibin\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.870985    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-binary-copy\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871018    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-config\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871074    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871104    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871150    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-slash\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871185    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871232    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03ef7e88-664f-413b-9424-c16a88d3b58c-data-volume\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871263    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-tmp\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871302    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-netd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871332    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-kubelet\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871381    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871417    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gs79\" (UniqueName: \"kubernetes.io/projected/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-kube-api-access-5gs79\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871458    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rtn\" (UniqueName: \"kubernetes.io/projected/19e0e140-91e4-4d92-b1ac-0ea2115e3467-kube-api-access-l4rtn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871496    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-bin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871540    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-host\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871573    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c09c0770-6cec-4827-9fb0-b07464653b27-tmp-dir\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871622    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rjb\" (UniqueName: \"kubernetes.io/projected/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-kube-api-access-x2rjb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871648    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-dbus\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871694    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871727    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-script-lib\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871774    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-config\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871808    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871840    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-system-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871872    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-cnibin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871901    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-socket-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871934    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkxh\" (UniqueName: \"kubernetes.io/projected/34c9bd1f-837b-4c0c-83da-3e5b09856707-kube-api-access-6bkxh\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.871972    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysconfig\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872003    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-systemd-units\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872036    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-iptables-alerter-script\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872091    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlch9\" (UniqueName: \"kubernetes.io/projected/46433966-ea95-45b6-9092-c2f73bd7cd53-kube-api-access-qlch9\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872121    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872155    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872188    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872217    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872249    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8pw\" (UniqueName: \"kubernetes.io/projected/3953b512-91d0-401c-8397-0b9314e7a87b-kube-api-access-rk8pw\") pod \"network-check-source-6f78d59f99-nwjwn\" (UID: \"3953b512-91d0-401c-8397-0b9314e7a87b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872277    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872308    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-kubernetes\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872342    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbs7\" (UniqueName: \"kubernetes.io/projected/3888b182-8fb4-49bd-9af3-873bcc3efa92-kube-api-access-cqbs7\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872372    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-conf-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872403    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-default-certificate\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872431    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxw6\" (UniqueName: \"kubernetes.io/projected/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-kube-api-access-4mxw6\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872463    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-systemd\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872498    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872529    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872559    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-socket-dir-parent\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872591    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6kk\" (UniqueName: \"kubernetes.io/projected/bad6344f-4c79-4140-abcc-2c4ae3d63049-kube-api-access-vc6kk\") pod \"kas-connection-checker-685895c7c9-tsmbq\" (UID: \"bad6344f-4c79-4140-abcc-2c4ae3d63049\") " pod="kube-system/kas-connection-checker-685895c7c9-tsmbq"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872621    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.872712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872629    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-tmp\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872856    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-netns\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872905    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-var-lib-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.872960    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-service-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873027    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-binary-copy\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873099    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-kubelet-config\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873106    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873141    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873629    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e5291295-5fd1-4e6e-b52b-c940c088168d-konnectivity-ca\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873648    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-config\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873650    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873683    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-trusted-ca\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873770    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-cni-binary-copy\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873819    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cedf131-a296-4792-8328-f3ad030d3003-multus-daemon-config\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873907    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-metrics-certs\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873927    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.873965    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e64a6-ecf0-471e-b130-15f85716445f-service-ca-bundle\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874094    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c09c0770-6cec-4827-9fb0-b07464653b27-tmp-dir\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874173    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e5291295-5fd1-4e6e-b52b-c940c088168d-agent-certs\") pod \"konnectivity-agent-bm5xm\" (UID: \"e5291295-5fd1-4e6e-b52b-c940c088168d\") " pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874205    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-iptables-alerter-script\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874210    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874259    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39facba4-434d-476e-9e84-a8388d78e4e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874493    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-config\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874504    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34c9bd1f-837b-4c0c-83da-3e5b09856707-cert\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874504    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8b43ebe-6a94-45f6-a613-0931f4607e9e-serviceca\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874687    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b41f9fc-afce-464d-a18c-330694f80140-config\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874740    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3888b182-8fb4-49bd-9af3-873bcc3efa92-metrics-certs\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874744    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-stats-auth\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874843    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c00596c9-ffa3-461b-81be-7315388deaf1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-744lb\" (UID: \"c00596c9-ffa3-461b-81be-7315388deaf1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-744lb"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874897    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-service-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874914    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19e0e140-91e4-4d92-b1ac-0ea2115e3467-ovnkube-script-lib\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874906    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874922    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874978    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03ef7e88-664f-413b-9424-c16a88d3b58c-data-volume\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.874999    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.875075    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/426e64a6-ecf0-471e-b130-15f85716445f-default-certificate\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.876021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.875138    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.918020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.917987    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6dv\" (UniqueName: \"kubernetes.io/projected/3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf-kube-api-access-6j6dv\") pod \"kas-connection-checker-685895c7c9-rsltv\" (UID: \"3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf\") " pod="kube-system/kas-connection-checker-685895c7c9-rsltv"
Nov 01 12:22:39.918020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.918019    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9jg\" (UniqueName: \"kubernetes.io/projected/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-kube-api-access-zb9jg\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:22:39.918266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.918248    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvmj\" (UniqueName: \"kubernetes.io/projected/3758a7d1-ccc2-4322-bd01-e742e5c11ff4-kube-api-access-kzvmj\") pod \"cluster-monitoring-operator-884576c57-jbpml\" (UID: \"3758a7d1-ccc2-4322-bd01-e742e5c11ff4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jbpml"
Nov 01 12:22:39.918315 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.918260    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8lp6\" (UniqueName: \"kubernetes.io/projected/d1efc2ea-a886-4c13-97a7-b1ddbd621feb-kube-api-access-m8lp6\") pod \"insights-operator-7c4986d449-rdmlg\" (UID: \"d1efc2ea-a886-4c13-97a7-b1ddbd621feb\") " pod="openshift-insights/insights-operator-7c4986d449-rdmlg"
Nov 01 12:22:39.919014 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.918986    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t59b\" (UniqueName: \"kubernetes.io/projected/f8b43ebe-6a94-45f6-a613-0931f4607e9e-kube-api-access-4t59b\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:22:39.919110 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.919025    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkmg\" (UniqueName: \"kubernetes.io/projected/c09c0770-6cec-4827-9fb0-b07464653b27-kube-api-access-fkkmg\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:22:39.919607 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.919588    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjsj\" (UniqueName: \"kubernetes.io/projected/9b41f9fc-afce-464d-a18c-330694f80140-kube-api-access-6rjsj\") pod \"console-operator-7cd56b8dbc-slc7h\" (UID: \"9b41f9fc-afce-464d-a18c-330694f80140\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:39.919685 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.919674    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rjm\" (UniqueName: \"kubernetes.io/projected/690056c7-6829-4caf-b59e-e2650d20bb47-kube-api-access-54rjm\") pod \"volume-data-source-validator-58c9f5f46d-j4vzd\" (UID: \"690056c7-6829-4caf-b59e-e2650d20bb47\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-j4vzd"
Nov 01 12:22:39.919815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.919725    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7csw\" (UniqueName: \"kubernetes.io/projected/ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c-kube-api-access-h7csw\") pod \"service-ca-operator-544d749c8c-sktlh\" (UID: \"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh"
Nov 01 12:22:39.919815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.919802    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s67j\" (UniqueName: \"kubernetes.io/projected/39facba4-434d-476e-9e84-a8388d78e4e9-kube-api-access-4s67j\") pod \"cluster-samples-operator-6ddb6c8b94-lq5xn\" (UID: \"39facba4-434d-476e-9e84-a8388d78e4e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lq5xn"
Nov 01 12:22:39.920281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.920256    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjtg\" (UniqueName: \"kubernetes.io/projected/4cf38c23-496f-4661-85ce-586416959d95-kube-api-access-jsjtg\") pod \"migrator-66df5d496b-twpxk\" (UID: \"4cf38c23-496f-4661-85ce-586416959d95\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-twpxk"
Nov 01 12:22:39.920341 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.920261    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwsm4\" (UniqueName: \"kubernetes.io/projected/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-kube-api-access-zwsm4\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.920642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.920626    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrg8\" (UniqueName: \"kubernetes.io/projected/03ef7e88-664f-413b-9424-c16a88d3b58c-kube-api-access-hhrg8\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.921061 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.921035    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wr4q\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.924362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.924346    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhf84\" (UniqueName: \"kubernetes.io/projected/b6cb3539-d27f-4954-9b4d-e730dfd5809d-kube-api-access-qhf84\") pod \"kas-connection-checker-685895c7c9-fg5h6\" (UID: \"b6cb3539-d27f-4954-9b4d-e730dfd5809d\") " pod="kube-system/kas-connection-checker-685895c7c9-fg5h6"
Nov 01 12:22:39.925378 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.925357    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rtn\" (UniqueName: \"kubernetes.io/projected/19e0e140-91e4-4d92-b1ac-0ea2115e3467-kube-api-access-l4rtn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.925455 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.925359    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqv4d\" (UniqueName: \"kubernetes.io/projected/426e64a6-ecf0-471e-b130-15f85716445f-kube-api-access-pqv4d\") pod \"router-default-79dd7d46d9-xfhjz\" (UID: \"426e64a6-ecf0-471e-b130-15f85716445f\") " pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:39.925659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.925641    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6kk\" (UniqueName: \"kubernetes.io/projected/bad6344f-4c79-4140-abcc-2c4ae3d63049-kube-api-access-vc6kk\") pod \"kas-connection-checker-685895c7c9-tsmbq\" (UID: \"bad6344f-4c79-4140-abcc-2c4ae3d63049\") " pod="kube-system/kas-connection-checker-685895c7c9-tsmbq"
Nov 01 12:22:39.925779 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.925761    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkxh\" (UniqueName: \"kubernetes.io/projected/34c9bd1f-837b-4c0c-83da-3e5b09856707-kube-api-access-6bkxh\") pod \"ingress-canary-x6gnr\" (UID: \"34c9bd1f-837b-4c0c-83da-3e5b09856707\") " pod="openshift-ingress-canary/ingress-canary-x6gnr"
Nov 01 12:22:39.925919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.925904    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbs7\" (UniqueName: \"kubernetes.io/projected/3888b182-8fb4-49bd-9af3-873bcc3efa92-kube-api-access-cqbs7\") pod \"network-metrics-daemon-p779s\" (UID: \"3888b182-8fb4-49bd-9af3-873bcc3efa92\") " pod="openshift-multus/network-metrics-daemon-p779s"
Nov 01 12:22:39.925952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.925910    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token\") pod \"image-registry-84fbdf67f8-6fmbg\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") " pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:39.926776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.926752    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rjb\" (UniqueName: \"kubernetes.io/projected/3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843-kube-api-access-x2rjb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xg58p\" (UID: \"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p"
Nov 01 12:22:39.926899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.926885    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmhd\" (UniqueName: \"kubernetes.io/projected/5cedf131-a296-4792-8328-f3ad030d3003-kube-api-access-7bmhd\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.927027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.927012    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8pw\" (UniqueName: \"kubernetes.io/projected/3953b512-91d0-401c-8397-0b9314e7a87b-kube-api-access-rk8pw\") pod \"network-check-source-6f78d59f99-nwjwn\" (UID: \"3953b512-91d0-401c-8397-0b9314e7a87b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nwjwn"
Nov 01 12:22:39.927391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.927376    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26dv\" (UniqueName: \"kubernetes.io/projected/47f53f35-b29e-4db5-91c8-0df05af362e7-kube-api-access-x26dv\") pod \"network-check-target-6zjj4\" (UID: \"47f53f35-b29e-4db5-91c8-0df05af362e7\") " pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:22:39.927724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.927704    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gs79\" (UniqueName: \"kubernetes.io/projected/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-kube-api-access-5gs79\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.927761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.927707    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlch9\" (UniqueName: \"kubernetes.io/projected/46433966-ea95-45b6-9092-c2f73bd7cd53-kube-api-access-qlch9\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.927801 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.927760    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxw6\" (UniqueName: \"kubernetes.io/projected/ee4ddc2b-35a6-4862-b64c-bb137efcdd69-kube-api-access-4mxw6\") pod \"dns-default-xm57l\" (UID: \"ee4ddc2b-35a6-4862-b64c-bb137efcdd69\") " pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:39.975333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975307    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-dbus\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:39.975423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975337    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-system-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975353    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-cnibin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975367    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-socket-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.975423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975391    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysconfig\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.975423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975413    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-systemd-units\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975432    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-system-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975441    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975438    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-cnibin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975464    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-socket-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975475    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-systemd-units\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975483    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-kubernetes\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975464    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-dbus\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975474    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysconfig\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975509    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975523    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-conf-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975528    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-kubernetes\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.975588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975569    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-conf-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975592    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-systemd\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975612    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975630    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-socket-dir-parent\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975648    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975686    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-cni-dir\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975688    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-netns\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975698    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-systemd\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975698    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-multus-socket-dir-parent\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975717    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-netns\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975716    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-var-lib-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975740    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-var-lib-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975744    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-kubelet-config\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975769    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-k8s-cni-cncf-io\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975785    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975788    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-etc-kubernetes\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975800    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/370a3bfd-1661-4e44-b5d5-5f6b83db1f6c-kubelet-config\") pod \"global-pull-secret-syncer-v7lhr\" (UID: \"370a3bfd-1661-4e44-b5d5-5f6b83db1f6c\") " pod="kube-system/global-pull-secret-syncer-v7lhr"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975808    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c09c0770-6cec-4827-9fb0-b07464653b27-hosts-file\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975811    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-k8s-cni-cncf-io\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975805    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-etc-kubernetes\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975845    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-hostroot\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975843    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c09c0770-6cec-4827-9fb0-b07464653b27-hosts-file\") pod \"node-resolver-kfdrq\" (UID: \"c09c0770-6cec-4827-9fb0-b07464653b27\") " pod="openshift-dns/node-resolver-kfdrq"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975867    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-multus-certs\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975893    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-multus-certs\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975899    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-hostroot\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975922    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03ef7e88-664f-413b-9424-c16a88d3b58c-crio-socket\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975943    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-lib-modules\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.975945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975960    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-kubelet\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975974    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.975994    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-registration-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976002    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03ef7e88-664f-413b-9424-c16a88d3b58c-crio-socket\") pod \"insights-runtime-extractor-9xgxt\" (UID: \"03ef7e88-664f-413b-9424-c16a88d3b58c\") " pod="openshift-insights/insights-runtime-extractor-9xgxt"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976017    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8b43ebe-6a94-45f6-a613-0931f4607e9e-host\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976022    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-kubelet\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976034    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-run-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976042    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-log-socket\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976077    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8b43ebe-6a94-45f6-a613-0931f4607e9e-host\") pod \"node-ca-fpq52\" (UID: \"f8b43ebe-6a94-45f6-a613-0931f4607e9e\") " pod="openshift-image-registry/node-ca-fpq52"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976079    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-registration-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976086    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976081    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-lib-modules\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976110    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-log-socket\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976118    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-conf\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976124    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976146    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-etc-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976167    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-run\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976188    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-ovn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976210    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-node-log\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976213    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-etc-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976218    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-sysctl-conf\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976235    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-var-lib-kubelet\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976246    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-run\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976256    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-bin\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976246    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-ovn\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976250    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-node-log\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976283    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-var-lib-kubelet\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976287    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-systemd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976296    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-bin\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976319    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-modprobe-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976341    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976358    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976377    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-sys-fs\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976399    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-netns\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976409    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-openvswitch\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976417    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-etc-modprobe-d\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976424    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-multus\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976319    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-run-systemd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976431    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976440    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-sys-fs\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976447    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-run-netns\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976451    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-sys\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976468    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-multus\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976480    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-sys\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976484    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-device-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976503    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-system-cni-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976519    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-host-slash\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976534    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976551    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cnibin\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976562    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffad4d99-42c6-4151-a4cf-e22b3ecd03e2-host-slash\") pod \"iptables-alerter-xwv48\" (UID: \"ffad4d99-42c6-4151-a4cf-e22b3ecd03e2\") " pod="openshift-network-operator/iptables-alerter-xwv48"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976535    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-system-cni-dir\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976568    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-slash\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976578    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-device-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976587    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46433966-ea95-45b6-9092-c2f73bd7cd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-czhh5\" (UID: \"46433966-ea95-45b6-9092-c2f73bd7cd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-czhh5"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976588    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/693b0401-a9f9-41a4-b2e8-d3e3d6f37477-cnibin\") pod \"multus-additional-cni-plugins-tg2vq\" (UID: \"693b0401-a9f9-41a4-b2e8-d3e3d6f37477\") " pod="openshift-multus/multus-additional-cni-plugins-tg2vq"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976602    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-netd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976603    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-slash\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976626    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-kubelet\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976659    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-bin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976673    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-host\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976677    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-kubelet\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976660    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19e0e140-91e4-4d92-b1ac-0ea2115e3467-host-cni-netd\") pod \"ovnkube-node-vrjcf\" (UID: \"19e0e140-91e4-4d92-b1ac-0ea2115e3467\") " pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976722    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cedf131-a296-4792-8328-f3ad030d3003-host-var-lib-cni-bin\") pod \"multus-lxjxw\" (UID: \"5cedf131-a296-4792-8328-f3ad030d3003\") " pod="openshift-multus/multus-lxjxw"
Nov 01 12:22:39.976832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:39.976752    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36-host\") pod \"tuned-dvz6d\" (UID: \"7a91a46a-94a6-466a-8c8f-8ae2f5ff4d36\") " pod="openshift-cluster-node-tuning-operator/tuned-dvz6d"
Nov 01 12:22:40.006029 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:40.006005    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:40.008311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:40.008295    5725 scope.go:122] "RemoveContainer" containerID="df2d582fa3b0ee2aa06cd65a280dac02c8b043b6d7a355aef887ebb44c671f0c"
Nov 01 12:22:40.681531 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:40.681458    5725 generic.go:356] "Generic (PLEG): container finished" podID="d1efc2ea-a886-4c13-97a7-b1ddbd621feb" containerID="a54f6554c65034f35162f1d6145dc39afe8de0006ae274b827a05812983f2333" exitCode=0
Nov 01 12:22:40.681960 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:40.681533    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rdmlg" event={"ID":"d1efc2ea-a886-4c13-97a7-b1ddbd621feb","Type":"ContainerDied","Data":"a54f6554c65034f35162f1d6145dc39afe8de0006ae274b827a05812983f2333"}
Nov 01 12:22:40.681960 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:40.681847    5725 scope.go:122] "RemoveContainer" containerID="a54f6554c65034f35162f1d6145dc39afe8de0006ae274b827a05812983f2333"
Nov 01 12:22:40.683512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:40.683495    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:22:40.683627 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:40.683608    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h" event={"ID":"9b41f9fc-afce-464d-a18c-330694f80140","Type":"ContainerStarted","Data":"24f9202bc4ab9826971a3569f29996418a982bba27b2a48a3a5d9c952327c816"}
Nov 01 12:22:40.728192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:40.728171    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:40.878720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:40.878689    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-slc7h"
Nov 01 12:22:41.008501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.008471    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:41.649394 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.649365    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:41.650552 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.650531    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79dd7d46d9-xfhjz"
Nov 01 12:22:41.690104 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.690043    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rdmlg" event={"ID":"d1efc2ea-a886-4c13-97a7-b1ddbd621feb","Type":"ContainerStarted","Data":"d439580a9908cbdb55443e1bc0f84326e1496ba75ec5d8a599f589d03c86c5eb"}
Nov 01 12:22:41.691659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.691635    5725 generic.go:356] "Generic (PLEG): container finished" podID="3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843" containerID="744941c4a9eca47c8f7844eb7dfe4453fa33d1c5657dd2163566adeb0dedf89c" exitCode=0
Nov 01 12:22:41.691789 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.691699    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p" event={"ID":"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843","Type":"ContainerDied","Data":"744941c4a9eca47c8f7844eb7dfe4453fa33d1c5657dd2163566adeb0dedf89c"}
Nov 01 12:22:41.692034 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.692010    5725 scope.go:122] "RemoveContainer" containerID="744941c4a9eca47c8f7844eb7dfe4453fa33d1c5657dd2163566adeb0dedf89c"
Nov 01 12:22:41.693318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.693292    5725 generic.go:356] "Generic (PLEG): container finished" podID="ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c" containerID="7e6625e5252700cbf041b994b922fe0636ff81bf4c7b78574534af120f8e8dc6" exitCode=0
Nov 01 12:22:41.693401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.693366    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh" event={"ID":"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c","Type":"ContainerDied","Data":"7e6625e5252700cbf041b994b922fe0636ff81bf4c7b78574534af120f8e8dc6"}
Nov 01 12:22:41.693822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:41.693801    5725 scope.go:122] "RemoveContainer" containerID="7e6625e5252700cbf041b994b922fe0636ff81bf4c7b78574534af120f8e8dc6"
Nov 01 12:22:42.699346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:42.699296    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xg58p" event={"ID":"3d2ec5b4-ed5e-49fa-ac4a-dbca006ed843","Type":"ContainerStarted","Data":"e74e62118ec28373a1bd3118d61b1fcde94843055766d5e88839e990676e4977"}
Nov 01 12:22:42.701110 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:42.701085    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-sktlh" event={"ID":"ad9f29a6-25a5-4dd3-bed6-ae0ca18cfe7c","Type":"ContainerStarted","Data":"b21463553d96ee883c709eb9a3c7a8c4b7ff6f3e64866be5eb07e71e4eee40f0"}
Nov 01 12:22:42.760798 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:42.760762    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:44.177716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:44.177673    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:44.796719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:44.796692    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:45.762876 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:45.762844    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xm57l"
Nov 01 12:22:46.810501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:46.810462    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:22:46.810974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:46.810720    5725 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 12:22:47.161432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.161359    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6zjj4"
Nov 01 12:22:47.623936 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.623902    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4pdcm"]
Nov 01 12:22:47.626475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.626452    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.725461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.725432    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 12:22:47.725642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.725477    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 12:22:47.725642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.725529    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ttg6g\""
Nov 01 12:22:47.725751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.725642    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 12:22:47.725751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.725708    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 12:22:47.735515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.735490    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-accelerators-collector-config\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.735642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.735527    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-textfile\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.735642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.735548    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5173ae11-40ca-47fe-a727-e39b21196422-metrics-client-ca\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.735642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.735608    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5173ae11-40ca-47fe-a727-e39b21196422-root\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.735755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.735673    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.735793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.735763    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr5rm\" (UniqueName: \"kubernetes.io/projected/5173ae11-40ca-47fe-a727-e39b21196422-kube-api-access-dr5rm\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.735793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.735786    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-tls\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.735884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.735805    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-wtmp\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.735884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.735823    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5173ae11-40ca-47fe-a727-e39b21196422-sys\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.836872 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.836832    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-textfile\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.836872 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.836880    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5173ae11-40ca-47fe-a727-e39b21196422-metrics-client-ca\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.836906    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5173ae11-40ca-47fe-a727-e39b21196422-root\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.836933    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.836972    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr5rm\" (UniqueName: \"kubernetes.io/projected/5173ae11-40ca-47fe-a727-e39b21196422-kube-api-access-dr5rm\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837010    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-tls\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837040    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-wtmp\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837043    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5173ae11-40ca-47fe-a727-e39b21196422-root\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837095    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5173ae11-40ca-47fe-a727-e39b21196422-sys\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837132    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5173ae11-40ca-47fe-a727-e39b21196422-sys\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837145    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-accelerators-collector-config\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837248    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-wtmp\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837418    5725 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 12:22:47.837898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837606    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-accelerators-collector-config\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837606    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5173ae11-40ca-47fe-a727-e39b21196422-metrics-client-ca\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.837898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.837798    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-textfile\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.839407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.839384    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.839537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.839519    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5173ae11-40ca-47fe-a727-e39b21196422-node-exporter-tls\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.893769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.893700    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr5rm\" (UniqueName: \"kubernetes.io/projected/5173ae11-40ca-47fe-a727-e39b21196422-kube-api-access-dr5rm\") pod \"node-exporter-4pdcm\" (UID: \"5173ae11-40ca-47fe-a727-e39b21196422\") " pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.938775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.938740    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4pdcm"
Nov 01 12:22:47.950636 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:22:47.950594    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5173ae11_40ca_47fe_a727_e39b21196422.slice/crio-a944b785ba55b23a05d3a47eb03c396892132a54d57ccde12c66bd7b6ec1cfe5 WatchSource:0}: Error finding container a944b785ba55b23a05d3a47eb03c396892132a54d57ccde12c66bd7b6ec1cfe5: Status 404 returned error can't find the container with id a944b785ba55b23a05d3a47eb03c396892132a54d57ccde12c66bd7b6ec1cfe5
Nov 01 12:22:47.952544 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:47.952524    5725 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:22:48.724371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:48.724333    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pdcm" event={"ID":"5173ae11-40ca-47fe-a727-e39b21196422","Type":"ContainerStarted","Data":"a944b785ba55b23a05d3a47eb03c396892132a54d57ccde12c66bd7b6ec1cfe5"}
Nov 01 12:22:49.731835 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:49.731741    5725 generic.go:356] "Generic (PLEG): container finished" podID="5173ae11-40ca-47fe-a727-e39b21196422" containerID="8bdca5d7cfd57cfddd17d916fa158240ada2393e44c49380f0dd07bf4ee55d98" exitCode=0
Nov 01 12:22:49.731835 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:49.731801    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pdcm" event={"ID":"5173ae11-40ca-47fe-a727-e39b21196422","Type":"ContainerDied","Data":"8bdca5d7cfd57cfddd17d916fa158240ada2393e44c49380f0dd07bf4ee55d98"}
Nov 01 12:22:50.737843 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:50.737800    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pdcm" event={"ID":"5173ae11-40ca-47fe-a727-e39b21196422","Type":"ContainerStarted","Data":"a76099f6482cec327ea997b6b80036980ffb44156ae0944f50471cb7219f838f"}
Nov 01 12:22:50.738267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:50.737861    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4pdcm" event={"ID":"5173ae11-40ca-47fe-a727-e39b21196422","Type":"ContainerStarted","Data":"9aa09fc987a63ae50b177ea0109e557a98dac96fe4cd8bb733b252ce41fb1799"}
Nov 01 12:22:50.848217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:50.848164    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4pdcm" podStartSLOduration=2.250272855 podStartE2EDuration="3.848150739s" totalImagesPullingTime="1.529213836s" totalInitContainerRuntime="68.664048ms" isStatefulPod=true podCreationTimestamp="2026-11-01 12:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:50.847401141 +0000 UTC m=+12.354592358" watchObservedRunningTime="2026-11-01 12:22:50.848150739 +0000 UTC m=+12.355341953"
Nov 01 12:22:51.608646 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:51.608608    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84fbdf67f8-6fmbg"]
Nov 01 12:22:51.612893 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:51.612868    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:22:51.630202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:22:51.630179    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bm5xm"
Nov 01 12:23:01.025791 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:23:01.025756    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:23:05.283119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:05.283087    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vrjcf"
Nov 01 12:23:16.628757 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.628691    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg" podUID="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" containerName="registry" containerID="cri-o://538ebd31136705a064ccb990ba4e4140e920d6e27ec38000aabda94621b09c02" gracePeriod=30
Nov 01 12:23:16.828530 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.828491    5725 generic.go:356] "Generic (PLEG): container finished" podID="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" containerID="538ebd31136705a064ccb990ba4e4140e920d6e27ec38000aabda94621b09c02" exitCode=0
Nov 01 12:23:16.828693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.828613    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg" event={"ID":"97f50f8c-9c08-406f-aba6-83fc9a2f75fd","Type":"ContainerDied","Data":"538ebd31136705a064ccb990ba4e4140e920d6e27ec38000aabda94621b09c02"}
Nov 01 12:23:16.891509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.891486    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:23:16.969279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.969248    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q\") pod \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") "
Nov 01 12:23:16.969453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.969285    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted\") pod \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") "
Nov 01 12:23:16.969453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.969310    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") pod \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") "
Nov 01 12:23:16.969453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.969354    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates\") pod \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") "
Nov 01 12:23:16.969453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.969388    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets\") pod \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") "
Nov 01 12:23:16.969672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.969453    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token\") pod \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") "
Nov 01 12:23:16.969672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.969496    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration\") pod \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") "
Nov 01 12:23:16.969672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.969612    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca\") pod \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\" (UID: \"97f50f8c-9c08-406f-aba6-83fc9a2f75fd\") "
Nov 01 12:23:16.970154 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.970111    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates" pod "97f50f8c-9c08-406f-aba6-83fc9a2f75fd" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:16.970270 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.970184    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca" pod "97f50f8c-9c08-406f-aba6-83fc9a2f75fd" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:16.972280 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.972249    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q" pod "97f50f8c-9c08-406f-aba6-83fc9a2f75fd" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd"). InnerVolumeSpecName "kube-api-access-9wr4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:23:16.972392 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.972346    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls" pod "97f50f8c-9c08-406f-aba6-83fc9a2f75fd" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:23:16.972464 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.972440    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token" pod "97f50f8c-9c08-406f-aba6-83fc9a2f75fd" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:23:16.972550 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.972523    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets" pod "97f50f8c-9c08-406f-aba6-83fc9a2f75fd" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:16.972650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.972584    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration" pod "97f50f8c-9c08-406f-aba6-83fc9a2f75fd" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:16.976642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:16.976619    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted" pod "97f50f8c-9c08-406f-aba6-83fc9a2f75fd" (UID: "97f50f8c-9c08-406f-aba6-83fc9a2f75fd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:23:17.071098 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.071063    5725 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-certificates\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:17.071098 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.071091    5725 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-installation-pull-secrets\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:17.071098 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.071101    5725 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-bound-sa-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:17.071314 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.071112    5725 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-image-registry-private-configuration\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:17.071314 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.071122    5725 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-trusted-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:17.071314 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.071131    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wr4q\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-kube-api-access-9wr4q\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:17.071314 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.071140    5725 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-ca-trust-extracted\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:17.071314 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.071149    5725 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97f50f8c-9c08-406f-aba6-83fc9a2f75fd-registry-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:17.833275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.833247    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg"
Nov 01 12:23:17.833704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.833243    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fbdf67f8-6fmbg" event={"ID":"97f50f8c-9c08-406f-aba6-83fc9a2f75fd","Type":"ContainerDied","Data":"7257c348c8d0768131c2427c447d061a1673df34fc8c14462bee77f3837dd452"}
Nov 01 12:23:17.833704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:17.833374    5725 scope.go:122] "RemoveContainer" containerID="538ebd31136705a064ccb990ba4e4140e920d6e27ec38000aabda94621b09c02"
Nov 01 12:23:18.033788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:18.033753    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84fbdf67f8-6fmbg"]
Nov 01 12:23:18.080899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:18.080867    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84fbdf67f8-6fmbg"]
Nov 01 12:23:18.616410 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:23:18.614623    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" path="/var/lib/kubelet/pods/97f50f8c-9c08-406f-aba6-83fc9a2f75fd/volumes"
Nov 01 12:23:31.026668 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:23:31.026628    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:24:01.026081 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:24:01.026021    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:24:31.025266 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:24:31.025187    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:25:01.027134 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:25:01.027094    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:25:31.026388 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:25:31.026341    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:26:01.025983 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:26:01.025896    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:26:31.026581 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:26:31.026541    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:27:01.025709 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:01.025668    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:27:18.963929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:18.963842    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"]
Nov 01 12:27:18.964514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:18.964328    5725 cpu_manager.go:414] "containerMap: removing container" podUID="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" containerName="registry"
Nov 01 12:27:18.964514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:18.964348    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" containerName="registry"
Nov 01 12:27:18.964514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:18.964436    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f50f8c-9c08-406f-aba6-83fc9a2f75fd" containerName="registry"
Nov 01 12:27:18.967672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:18.967648    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.019560 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.019523    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 12:27:19.020351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.020330    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"]
Nov 01 12:27:19.042030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.041996    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.042030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.042032    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.042246 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.042090    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wz89\" (UniqueName: \"kubernetes.io/projected/ce09757e-992e-463d-a9bf-b6c807827d9d-kube-api-access-5wz89\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.066676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.066646    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 12:27:19.117465 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.117439    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-g55fb\""
Nov 01 12:27:19.142874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.142847    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.143043 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.142877    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.143043 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.142903    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wz89\" (UniqueName: \"kubernetes.io/projected/ce09757e-992e-463d-a9bf-b6c807827d9d-kube-api-access-5wz89\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.143292 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.143268    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.143355 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.143297    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.196720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.196681    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wz89\" (UniqueName: \"kubernetes.io/projected/ce09757e-992e-463d-a9bf-b6c807827d9d-kube-api-access-5wz89\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.282914 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.282881    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:27:19.653401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.653316    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"]
Nov 01 12:27:19.696502 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:19.696443    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce09757e_992e_463d_a9bf_b6c807827d9d.slice/crio-bb91edb4ecdb3a65ff6556bac71eef3ce63ed135a604a61edece93a65d150db1 WatchSource:0}: Error finding container bb91edb4ecdb3a65ff6556bac71eef3ce63ed135a604a61edece93a65d150db1: Status 404 returned error can't find the container with id bb91edb4ecdb3a65ff6556bac71eef3ce63ed135a604a61edece93a65d150db1
Nov 01 12:27:19.930918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.930845    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"]
Nov 01 12:27:19.935288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.935271    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:19.986860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:19.986829    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"]
Nov 01 12:27:20.031715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.031688    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"]
Nov 01 12:27:20.034926 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.034911    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.038511 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.038490    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-wspv9\""
Nov 01 12:27:20.050764 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.050743    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.050863 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.050798    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tgsh\" (UniqueName: \"kubernetes.io/projected/790b64a7-9958-424d-b462-a29c3f97d8de-kube-api-access-5tgsh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.050899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.050866    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.084491 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.084472    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2bdwr\""
Nov 01 12:27:20.088915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.088892    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"]
Nov 01 12:27:20.140448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.140409    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"]
Nov 01 12:27:20.143733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.143711    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.151237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.151218    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.151365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.151256    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.151365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.151276    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.151528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.151509    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.151588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.151569    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tgsh\" (UniqueName: \"kubernetes.io/projected/790b64a7-9958-424d-b462-a29c3f97d8de-kube-api-access-5tgsh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.151648 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.151624    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvw7\" (UniqueName: \"kubernetes.io/projected/90780cda-beee-4fe4-b767-0f28893febe4-kube-api-access-jmvw7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.151692 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.151683    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.151733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.151635    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.192873 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.192807    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-25jxc\""
Nov 01 12:27:20.200414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.200382    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"]
Nov 01 12:27:20.205695 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.205660    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tgsh\" (UniqueName: \"kubernetes.io/projected/790b64a7-9958-424d-b462-a29c3f97d8de-kube-api-access-5tgsh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.230677 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.230645    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"]
Nov 01 12:27:20.233993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.233976    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.251291 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.251271    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:27:20.252551 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.252534    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.252679 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.252564    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvw7\" (UniqueName: \"kubernetes.io/projected/90780cda-beee-4fe4-b767-0f28893febe4-kube-api-access-jmvw7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.252679 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.252589    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.252792 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.252672    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.252792 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.252764    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.252867 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.252811    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8j6\" (UniqueName: \"kubernetes.io/projected/49703698-166d-4371-8556-da90a48de9d8-kube-api-access-6m8j6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.252973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.252955    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.253103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.253089    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.287717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.287681    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"]
Nov 01 12:27:20.293288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.293258    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-62mbb\""
Nov 01 12:27:20.307713 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.307686    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvw7\" (UniqueName: \"kubernetes.io/projected/90780cda-beee-4fe4-b767-0f28893febe4-kube-api-access-jmvw7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.349850 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.349819    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:27:20.353599 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.353569    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.353737 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.353613    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.353737 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.353676    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.354237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.353776    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8j6\" (UniqueName: \"kubernetes.io/projected/49703698-166d-4371-8556-da90a48de9d8-kube-api-access-6m8j6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.354237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.353817    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.354237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.353870    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhcp\" (UniqueName: \"kubernetes.io/projected/757d9150-058a-4d63-b9e6-51359af63684-kube-api-access-lfhcp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.354237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.353903    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.354376 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.354245    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.410318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.410290    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8j6\" (UniqueName: \"kubernetes.io/projected/49703698-166d-4371-8556-da90a48de9d8-kube-api-access-6m8j6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.455265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.455180    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhcp\" (UniqueName: \"kubernetes.io/projected/757d9150-058a-4d63-b9e6-51359af63684-kube-api-access-lfhcp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.455265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.455223    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.455265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.455248    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.455678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.455653    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.455733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.455658    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.460084 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.460069    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:27:20.509860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.509831    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhcp\" (UniqueName: \"kubernetes.io/projected/757d9150-058a-4d63-b9e6-51359af63684-kube-api-access-lfhcp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.549640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.549610    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:27:20.620584 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.620555    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"]
Nov 01 12:27:20.629270 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:20.629240    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790b64a7_9958_424d_b462_a29c3f97d8de.slice/crio-c601c801746ade78a0e62ee34866bde92b484b1a8436cea80f05038b3942c662 WatchSource:0}: Error finding container c601c801746ade78a0e62ee34866bde92b484b1a8436cea80f05038b3942c662: Status 404 returned error can't find the container with id c601c801746ade78a0e62ee34866bde92b484b1a8436cea80f05038b3942c662
Nov 01 12:27:20.664295 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.664254    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2" event={"ID":"790b64a7-9958-424d-b462-a29c3f97d8de","Type":"ContainerStarted","Data":"c601c801746ade78a0e62ee34866bde92b484b1a8436cea80f05038b3942c662"}
Nov 01 12:27:20.665640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.665595    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82" event={"ID":"ce09757e-992e-463d-a9bf-b6c807827d9d","Type":"ContainerStarted","Data":"bb91edb4ecdb3a65ff6556bac71eef3ce63ed135a604a61edece93a65d150db1"}
Nov 01 12:27:20.730461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.730432    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"]
Nov 01 12:27:20.739374 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:20.739349    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90780cda_beee_4fe4_b767_0f28893febe4.slice/crio-0217ec8a7a4e1691e9518d8e6d3b3e460b9add0a18883bd4537fafd4e25a9369 WatchSource:0}: Error finding container 0217ec8a7a4e1691e9518d8e6d3b3e460b9add0a18883bd4537fafd4e25a9369: Status 404 returned error can't find the container with id 0217ec8a7a4e1691e9518d8e6d3b3e460b9add0a18883bd4537fafd4e25a9369
Nov 01 12:27:20.826769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.826719    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"]
Nov 01 12:27:20.835908 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:20.835880    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49703698_166d_4371_8556_da90a48de9d8.slice/crio-d3baa21bdc6474ee4e36448e7d881795a5eb20eabfc2f7c8969b3ecf79b0b158 WatchSource:0}: Error finding container d3baa21bdc6474ee4e36448e7d881795a5eb20eabfc2f7c8969b3ecf79b0b158: Status 404 returned error can't find the container with id d3baa21bdc6474ee4e36448e7d881795a5eb20eabfc2f7c8969b3ecf79b0b158
Nov 01 12:27:20.918044 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:20.918006    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"]
Nov 01 12:27:20.927388 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:20.927360    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757d9150_058a_4d63_b9e6_51359af63684.slice/crio-0897f8b7e75948982186bd40d8b19f6fb7aab4adc44217768191772ed7b401af WatchSource:0}: Error finding container 0897f8b7e75948982186bd40d8b19f6fb7aab4adc44217768191772ed7b401af: Status 404 returned error can't find the container with id 0897f8b7e75948982186bd40d8b19f6fb7aab4adc44217768191772ed7b401af
Nov 01 12:27:21.030349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.030280    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"]
Nov 01 12:27:21.035201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.035177    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.086920 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.086887    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-cf5bx\""
Nov 01 12:27:21.087934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.087901    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"]
Nov 01 12:27:21.160493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.160453    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.160493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.160490    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.160707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.160607    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crrw8\" (UniqueName: \"kubernetes.io/projected/03aea48c-4114-42da-b622-0bd5f8e7442f-kube-api-access-crrw8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.261329 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.261293    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.261487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.261342    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.261487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.261396    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crrw8\" (UniqueName: \"kubernetes.io/projected/03aea48c-4114-42da-b622-0bd5f8e7442f-kube-api-access-crrw8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.261746 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.261720    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.261821 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.261758    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.315423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.315354    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crrw8\" (UniqueName: \"kubernetes.io/projected/03aea48c-4114-42da-b622-0bd5f8e7442f-kube-api-access-crrw8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.354255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.354223    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:27:21.672016 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.671923    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk" event={"ID":"49703698-166d-4371-8556-da90a48de9d8","Type":"ContainerStarted","Data":"d3baa21bdc6474ee4e36448e7d881795a5eb20eabfc2f7c8969b3ecf79b0b158"}
Nov 01 12:27:21.673306 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.673275    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m" event={"ID":"90780cda-beee-4fe4-b767-0f28893febe4","Type":"ContainerStarted","Data":"0217ec8a7a4e1691e9518d8e6d3b3e460b9add0a18883bd4537fafd4e25a9369"}
Nov 01 12:27:21.674779 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.674746    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6" event={"ID":"757d9150-058a-4d63-b9e6-51359af63684","Type":"ContainerStarted","Data":"0897f8b7e75948982186bd40d8b19f6fb7aab4adc44217768191772ed7b401af"}
Nov 01 12:27:21.727359 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:21.727323    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"]
Nov 01 12:27:21.737656 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:21.737615    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03aea48c_4114_42da_b622_0bd5f8e7442f.slice/crio-0b770546b388db9830c9d4002c2fdd619e5b3bde86548c850bf91f79ade175e8 WatchSource:0}: Error finding container 0b770546b388db9830c9d4002c2fdd619e5b3bde86548c850bf91f79ade175e8: Status 404 returned error can't find the container with id 0b770546b388db9830c9d4002c2fdd619e5b3bde86548c850bf91f79ade175e8
Nov 01 12:27:22.684742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:22.684688    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk" event={"ID":"03aea48c-4114-42da-b622-0bd5f8e7442f","Type":"ContainerStarted","Data":"0b770546b388db9830c9d4002c2fdd619e5b3bde86548c850bf91f79ade175e8"}
Nov 01 12:27:23.116940 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.116905    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"]
Nov 01 12:27:23.121296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.121265    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.180985 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.180943    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn269\" (UniqueName: \"kubernetes.io/projected/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-kube-api-access-cn269\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.181137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.181003    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.181137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.181062    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.219820 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.219789    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-xpscr\""
Nov 01 12:27:23.223078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.222972    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"]
Nov 01 12:27:23.282538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.282501    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn269\" (UniqueName: \"kubernetes.io/projected/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-kube-api-access-cn269\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.282715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.282555    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.282715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.282599    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.283090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.283065    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.283626 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.283600    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.338448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.338414    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn269\" (UniqueName: \"kubernetes.io/projected/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-kube-api-access-cn269\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.444206 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.444114    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:27:23.877120 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:23.877087    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"]
Nov 01 12:27:23.906922 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:23.906884    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3ac5ee_f5a4_4bc4_84fc_f7d1e1a52eb5.slice/crio-3fee40f7d4224c548029592bfc0fa5f8c5e5b972c91568c80b588a600c28ada7 WatchSource:0}: Error finding container 3fee40f7d4224c548029592bfc0fa5f8c5e5b972c91568c80b588a600c28ada7: Status 404 returned error can't find the container with id 3fee40f7d4224c548029592bfc0fa5f8c5e5b972c91568c80b588a600c28ada7
Nov 01 12:27:24.703860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:24.703778    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t" event={"ID":"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5","Type":"ContainerStarted","Data":"3fee40f7d4224c548029592bfc0fa5f8c5e5b972c91568c80b588a600c28ada7"}
Nov 01 12:27:25.252374 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.252338    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"]
Nov 01 12:27:25.256753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.256656    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.306659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.306426    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.306659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.306475    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.306659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.306603    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkbb\" (UniqueName: \"kubernetes.io/projected/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-kube-api-access-xzkbb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.309797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.309769    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"]
Nov 01 12:27:25.408717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.407492    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkbb\" (UniqueName: \"kubernetes.io/projected/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-kube-api-access-xzkbb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.408717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.407549    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.408717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.407578    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.408717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.408017    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.408717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.408658    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.463347 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.463244    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkbb\" (UniqueName: \"kubernetes.io/projected/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-kube-api-access-xzkbb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.579646 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.579343    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:27:25.978346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:25.978309    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"]
Nov 01 12:27:26.040513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.038781    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"]
Nov 01 12:27:26.043923 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.043895    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.093697 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.093631    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bc8bt\""
Nov 01 12:27:26.097485 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.097426    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 12:27:26.114844 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.114811    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.114996 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.114855    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.114996 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.114939    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2nx8\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-kube-api-access-p2nx8\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.114996 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.114989    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.115238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.115031    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.115238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.115102    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.115238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.115178    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec5523e-d2ad-414c-9186-003789159540-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.115238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.115206    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec5523e-d2ad-414c-9186-003789159540-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.115421 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.115239    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.117476 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.117451    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"]
Nov 01 12:27:26.216087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.216033    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.216087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.216114    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.216087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.216142    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.216535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.216178    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2nx8\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-kube-api-access-p2nx8\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.216535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.216224    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.216535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.216261    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.216535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.216310    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.216535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.216341    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec5523e-d2ad-414c-9186-003789159540-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.216535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.216369    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec5523e-d2ad-414c-9186-003789159540-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.217528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.217462    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.217995 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.217931    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.218319 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.218197    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.218446 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.218422    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.219364 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.219300    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec5523e-d2ad-414c-9186-003789159540-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.226436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.226409    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.227785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.227741    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec5523e-d2ad-414c-9186-003789159540-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.280100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.279977    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.286777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.286687    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2nx8\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-kube-api-access-p2nx8\") pod \"maas-default-gateway-openshift-default-7df854bf67-h6tl4\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.330596 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.328902    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"]
Nov 01 12:27:26.334726 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.333820    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.362079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.359838    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:27:26.384009 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.383833    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-t44xg\""
Nov 01 12:27:26.389825 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.389799    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"]
Nov 01 12:27:26.417933 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.417876    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.418130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.417942    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.418130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.417989    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.418130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.418027    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.418130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.418111    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.418356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.418141    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.418356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.418164    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.418356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.418192    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9zz\" (UniqueName: \"kubernetes.io/projected/de6b7627-2b0c-416b-b3fe-6f6d03e09089-kube-api-access-ld9zz\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.418356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.418223    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.519108 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519041    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.519281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519121    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.519281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519180    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.519281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519208    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.519281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519228    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.519281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519255    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9zz\" (UniqueName: \"kubernetes.io/projected/de6b7627-2b0c-416b-b3fe-6f6d03e09089-kube-api-access-ld9zz\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.519548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519288    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.519548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519352    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.519548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519375    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.520838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.519777    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.520838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.520024    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.520838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.520039    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.520838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.520805    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.521156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.521121    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.535005 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.534941    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.549187 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.549082    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.586010 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.585762    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9zz\" (UniqueName: \"kubernetes.io/projected/de6b7627-2b0c-416b-b3fe-6f6d03e09089-kube-api-access-ld9zz\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.586857 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.586779    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6b7627-2b0c-416b-b3fe-6f6d03e09089-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl\" (UID: \"de6b7627-2b0c-416b-b3fe-6f6d03e09089\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.652711 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.652411    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:27:26.746522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.742185    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw" event={"ID":"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d","Type":"ContainerStarted","Data":"aada9f758e60ba02286592c3b054c7b62d027c139349e4bffe43e0c4dd8fab21"}
Nov 01 12:27:26.780137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:26.779935    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"]
Nov 01 12:27:26.797160 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:26.797103    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec5523e_d2ad_414c_9186_003789159540.slice/crio-439528ad00db5af49d0b4bdc9607d2fe468d87080c893f9c4a414759e549f0f8 WatchSource:0}: Error finding container 439528ad00db5af49d0b4bdc9607d2fe468d87080c893f9c4a414759e549f0f8: Status 404 returned error can't find the container with id 439528ad00db5af49d0b4bdc9607d2fe468d87080c893f9c4a414759e549f0f8
Nov 01 12:27:27.062487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.062356    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"]
Nov 01 12:27:27.073789 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:27.073756    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6b7627_2b0c_416b_b3fe_6f6d03e09089.slice/crio-c27f73c77b07ed6489fc5ede03db00ec0447b9ff5adce45fe4be0291c16579df WatchSource:0}: Error finding container c27f73c77b07ed6489fc5ede03db00ec0447b9ff5adce45fe4be0291c16579df: Status 404 returned error can't find the container with id c27f73c77b07ed6489fc5ede03db00ec0447b9ff5adce45fe4be0291c16579df
Nov 01 12:27:27.387692 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.387608    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"]
Nov 01 12:27:27.392015 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.391988    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.415410 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.415362    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"]
Nov 01 12:27:27.431058 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.431008    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.431229 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.431114    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tm79\" (UniqueName: \"kubernetes.io/projected/f83f11a3-9457-4766-b72d-7988cda871cf-kube-api-access-8tm79\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.431351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.431222    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.437724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.437696    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.439602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.439579    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"]
Nov 01 12:27:27.443651 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.443630    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.448011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.447989    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"]
Nov 01 12:27:27.470093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.468789    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"]
Nov 01 12:27:27.475176 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.475152    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.490192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.490165    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"]
Nov 01 12:27:27.532081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.530957    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"]
Nov 01 12:27:27.532779 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.532744    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.532883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.532790    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tm79\" (UniqueName: \"kubernetes.io/projected/f83f11a3-9457-4766-b72d-7988cda871cf-kube-api-access-8tm79\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.532883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.532821    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.533000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.532884    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.533000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.532914    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.533000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.532951    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.533000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.532989    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.533292 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.533019    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.534412 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.533494    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.534812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.534566    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59t7p\" (UniqueName: \"kubernetes.io/projected/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-kube-api-access-59t7p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.534812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.534614    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.534812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.534650    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz8qq\" (UniqueName: \"kubernetes.io/projected/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-kube-api-access-zz8qq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.534812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.534674    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klrts\" (UniqueName: \"kubernetes.io/projected/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-kube-api-access-klrts\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.535266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.535180    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.545358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.545221    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"]
Nov 01 12:27:27.589492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.589371    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tm79\" (UniqueName: \"kubernetes.io/projected/f83f11a3-9457-4766-b72d-7988cda871cf-kube-api-access-8tm79\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.635510    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.635586    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.635614    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.635652    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59t7p\" (UniqueName: \"kubernetes.io/projected/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-kube-api-access-59t7p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.635686    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.635733    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz8qq\" (UniqueName: \"kubernetes.io/projected/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-kube-api-access-zz8qq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.635759    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klrts\" (UniqueName: \"kubernetes.io/projected/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-kube-api-access-klrts\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.635818    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.635868    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.636255    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.636477    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.636646    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.636903    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.637541    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.638134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.638009    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.694094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.694061    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz8qq\" (UniqueName: \"kubernetes.io/projected/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-kube-api-access-zz8qq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:27.696435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.696333    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59t7p\" (UniqueName: \"kubernetes.io/projected/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-kube-api-access-59t7p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.699720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.699693    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klrts\" (UniqueName: \"kubernetes.io/projected/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-kube-api-access-klrts\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.723546 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.723335    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:27:27.752319 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.751853    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl" event={"ID":"de6b7627-2b0c-416b-b3fe-6f6d03e09089","Type":"ContainerStarted","Data":"c27f73c77b07ed6489fc5ede03db00ec0447b9ff5adce45fe4be0291c16579df"}
Nov 01 12:27:27.759991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.759951    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" event={"ID":"9ec5523e-d2ad-414c-9186-003789159540","Type":"ContainerStarted","Data":"439528ad00db5af49d0b4bdc9607d2fe468d87080c893f9c4a414759e549f0f8"}
Nov 01 12:27:27.763359 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.763332    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:27:27.785214 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.785185    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:27:27.806810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:27.806783    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:27:28.183131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:28.182948    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"]
Nov 01 12:27:28.214817 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:28.214771    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83f11a3_9457_4766_b72d_7988cda871cf.slice/crio-4fae626bbc44295a1ac7fb7b93682e7f71f6e4bd86dd34136fd70a8f3833272d WatchSource:0}: Error finding container 4fae626bbc44295a1ac7fb7b93682e7f71f6e4bd86dd34136fd70a8f3833272d: Status 404 returned error can't find the container with id 4fae626bbc44295a1ac7fb7b93682e7f71f6e4bd86dd34136fd70a8f3833272d
Nov 01 12:27:28.235157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:28.235111    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"]
Nov 01 12:27:28.238353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:28.238292    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"]
Nov 01 12:27:28.246507 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:28.246463    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eedbaf1_cd65_4c1a_9e38_99fb7fc36aba.slice/crio-a86d11e25f369c14181ad428ad0d3b999b98c9a07c12694a6ec966b419d63366 WatchSource:0}: Error finding container a86d11e25f369c14181ad428ad0d3b999b98c9a07c12694a6ec966b419d63366: Status 404 returned error can't find the container with id a86d11e25f369c14181ad428ad0d3b999b98c9a07c12694a6ec966b419d63366
Nov 01 12:27:28.250041 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:28.250011    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d1c1b7_fdf1_42c4_996f_be697a1a3cdd.slice/crio-95995ad1aa51d90c124023cab4a20b0c59e5c6da5b6cb9c2d166bead18ae04a2 WatchSource:0}: Error finding container 95995ad1aa51d90c124023cab4a20b0c59e5c6da5b6cb9c2d166bead18ae04a2: Status 404 returned error can't find the container with id 95995ad1aa51d90c124023cab4a20b0c59e5c6da5b6cb9c2d166bead18ae04a2
Nov 01 12:27:28.290268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:28.290145    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"]
Nov 01 12:27:28.306035 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:27:28.306000    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f0a0c8_7e3b_45b1_8c0c_4b73676f1cca.slice/crio-768c1fcb4a4aca04768baa3ebee4089dca293668070ca720282eca061f2b92c9 WatchSource:0}: Error finding container 768c1fcb4a4aca04768baa3ebee4089dca293668070ca720282eca061f2b92c9: Status 404 returned error can't find the container with id 768c1fcb4a4aca04768baa3ebee4089dca293668070ca720282eca061f2b92c9
Nov 01 12:27:28.781006 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:28.780861    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2" event={"ID":"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca","Type":"ContainerStarted","Data":"768c1fcb4a4aca04768baa3ebee4089dca293668070ca720282eca061f2b92c9"}
Nov 01 12:27:28.787828 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:28.787790    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj" event={"ID":"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd","Type":"ContainerStarted","Data":"95995ad1aa51d90c124023cab4a20b0c59e5c6da5b6cb9c2d166bead18ae04a2"}
Nov 01 12:27:28.796602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:28.796477    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw" event={"ID":"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba","Type":"ContainerStarted","Data":"a86d11e25f369c14181ad428ad0d3b999b98c9a07c12694a6ec966b419d63366"}
Nov 01 12:27:28.800043 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:28.799975    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24" event={"ID":"f83f11a3-9457-4766-b72d-7988cda871cf","Type":"ContainerStarted","Data":"4fae626bbc44295a1ac7fb7b93682e7f71f6e4bd86dd34136fd70a8f3833272d"}
Nov 01 12:27:31.069409 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:31.069322    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:27:34.056133 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:34.055501    5725 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage3904850117/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/b8/b85d11a7271770efa17b37218f06dad95544e470a0319eea3f5d383b0fdd060a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261101T122732Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=9455714f20230456e03723ab3217a5e4080c92f936ed9cde4a6fd5cf2b3dcc83&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793536952~hmac=d88877bd1acaafac4f27378823cafd42dd57dbb4299e5797b203eff2637cf156\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" image="registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
Nov 01 12:27:34.056133 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:34.055661    5725 kuberuntime_manager.go:1707] "Unhandled Error" err="container istio-proxy start failed in pod maas-default-gateway-openshift-default-7df854bf67-h6tl4_openshift-ingress(9ec5523e-d2ad-414c-9186-003789159540): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage3904850117/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/b8/b85d11a7271770efa17b37218f06dad95544e470a0319eea3f5d383b0fdd060a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261101T122732Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=9455714f20230456e03723ab3217a5e4080c92f936ed9cde4a6fd5cf2b3dcc83&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793536952~hmac=d88877bd1acaafac4f27378823cafd42dd57dbb4299e5797b203eff2637cf156\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" logger="UnhandledError"
Nov 01 12:27:34.056133 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:34.055749    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage3904850117/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/b8/b85d11a7271770efa17b37218f06dad95544e470a0319eea3f5d383b0fdd060a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261101T122732Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=9455714f20230456e03723ab3217a5e4080c92f936ed9cde4a6fd5cf2b3dcc83&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793536952~hmac=d88877bd1acaafac4f27378823cafd42dd57dbb4299e5797b203eff2637cf156\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" podUID="9ec5523e-d2ad-414c-9186-003789159540"
Nov 01 12:27:34.899725 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:34.899411    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage3904850117/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/b8/b85d11a7271770efa17b37218f06dad95544e470a0319eea3f5d383b0fdd060a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261101T122732Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=9455714f20230456e03723ab3217a5e4080c92f936ed9cde4a6fd5cf2b3dcc83&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793536952~hmac=d88877bd1acaafac4f27378823cafd42dd57dbb4299e5797b203eff2637cf156\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" podUID="9ec5523e-d2ad-414c-9186-003789159540"
Nov 01 12:27:35.460979 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:35.460919    5725 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage435757631/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/b8/b85d11a7271770efa17b37218f06dad95544e470a0319eea3f5d383b0fdd060a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261101T122734Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=7fe0790540bb34fa7b91abdb975147f3d9ffe64c022a2f9ea743cc6e834c028b&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793536954~hmac=165a64bc6bf4d9b2f74e05f4a7131edbdad067394e07ac9b708567978c78d7ca\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" image="registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55"
Nov 01 12:27:35.461510 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:35.461282    5725 kuberuntime_manager.go:1707] "Unhandled Error" err="container istio-proxy start failed in pod openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl_openshift-ingress(de6b7627-2b0c-416b-b3fe-6f6d03e09089): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage435757631/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/b8/b85d11a7271770efa17b37218f06dad95544e470a0319eea3f5d383b0fdd060a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261101T122734Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=7fe0790540bb34fa7b91abdb975147f3d9ffe64c022a2f9ea743cc6e834c028b&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793536954~hmac=165a64bc6bf4d9b2f74e05f4a7131edbdad067394e07ac9b708567978c78d7ca\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" logger="UnhandledError"
Nov 01 12:27:35.461510 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:35.461353    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage435757631/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/b8/b85d11a7271770efa17b37218f06dad95544e470a0319eea3f5d383b0fdd060a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261101T122734Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=7fe0790540bb34fa7b91abdb975147f3d9ffe64c022a2f9ea743cc6e834c028b&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793536954~hmac=165a64bc6bf4d9b2f74e05f4a7131edbdad067394e07ac9b708567978c78d7ca\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl" podUID="de6b7627-2b0c-416b-b3fe-6f6d03e09089"
Nov 01 12:27:35.916682 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:27:35.916632    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage435757631/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/b8/b85d11a7271770efa17b37218f06dad95544e470a0319eea3f5d383b0fdd060a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261101T122734Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=7fe0790540bb34fa7b91abdb975147f3d9ffe64c022a2f9ea743cc6e834c028b&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793536954~hmac=165a64bc6bf4d9b2f74e05f4a7131edbdad067394e07ac9b708567978c78d7ca\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl" podUID="de6b7627-2b0c-416b-b3fe-6f6d03e09089"
Nov 01 12:27:38.676406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:38.676383    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:27:38.691296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:38.691273    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:27:38.953984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:38.953942    5725 generic.go:356] "Generic (PLEG): container finished" podID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerID="cf2e27f4bda094e70ab6b9e07c07fa5ad0bc03b6db175b16c3a4d54b23160926" exitCode=0
Nov 01 12:27:38.954208 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:38.954029    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82" event={"ID":"ce09757e-992e-463d-a9bf-b6c807827d9d","Type":"ContainerDied","Data":"cf2e27f4bda094e70ab6b9e07c07fa5ad0bc03b6db175b16c3a4d54b23160926"}
Nov 01 12:27:40.976454 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:40.976007    5725 generic.go:356] "Generic (PLEG): container finished" podID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerID="fc92f697adca878ed7986e56f13dd0387a49b18e314cd3864afe89cb7008c2a3" exitCode=0
Nov 01 12:27:40.976454 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:40.976156    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw" event={"ID":"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d","Type":"ContainerDied","Data":"fc92f697adca878ed7986e56f13dd0387a49b18e314cd3864afe89cb7008c2a3"}
Nov 01 12:27:41.985152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:41.985108    5725 generic.go:356] "Generic (PLEG): container finished" podID="757d9150-058a-4d63-b9e6-51359af63684" containerID="74bb87e6cbf72e2f04589778ad3f42449b8342835357c59c3965571cfbc6fa77" exitCode=0
Nov 01 12:27:41.985623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:41.985272    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6" event={"ID":"757d9150-058a-4d63-b9e6-51359af63684","Type":"ContainerDied","Data":"74bb87e6cbf72e2f04589778ad3f42449b8342835357c59c3965571cfbc6fa77"}
Nov 01 12:27:41.989618 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:41.989471    5725 generic.go:356] "Generic (PLEG): container finished" podID="49703698-166d-4371-8556-da90a48de9d8" containerID="b467c039994ed7c1bf07bc0502746db3d7b14e0178cb088384b5e290acc33068" exitCode=0
Nov 01 12:27:41.989618 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:41.989563    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk" event={"ID":"49703698-166d-4371-8556-da90a48de9d8","Type":"ContainerDied","Data":"b467c039994ed7c1bf07bc0502746db3d7b14e0178cb088384b5e290acc33068"}
Nov 01 12:27:42.996705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:42.996677    5725 generic.go:356] "Generic (PLEG): container finished" podID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerID="9c00c9fdca6fee6527ef1d7a7d0ebde65223e2fcbe9e8f7f1f81a810232f9ecf" exitCode=0
Nov 01 12:27:42.997071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:42.996750    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2" event={"ID":"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca","Type":"ContainerDied","Data":"9c00c9fdca6fee6527ef1d7a7d0ebde65223e2fcbe9e8f7f1f81a810232f9ecf"}
Nov 01 12:27:43.001714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:43.001123    5725 generic.go:356] "Generic (PLEG): container finished" podID="90780cda-beee-4fe4-b767-0f28893febe4" containerID="d04f655803eb7113558b5de3f24d0984e6a1c5c0616fe3998f44b055ec7a5afe" exitCode=0
Nov 01 12:27:43.001714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:43.001246    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m" event={"ID":"90780cda-beee-4fe4-b767-0f28893febe4","Type":"ContainerDied","Data":"d04f655803eb7113558b5de3f24d0984e6a1c5c0616fe3998f44b055ec7a5afe"}
Nov 01 12:27:43.003521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:43.003495    5725 generic.go:356] "Generic (PLEG): container finished" podID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerID="5453c4eba68d1aefc588d90a4fb7744ba29014da8bb35ab1cf19e176d230670e" exitCode=0
Nov 01 12:27:43.003615 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:43.003599    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw" event={"ID":"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba","Type":"ContainerDied","Data":"5453c4eba68d1aefc588d90a4fb7744ba29014da8bb35ab1cf19e176d230670e"}
Nov 01 12:27:43.005753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:43.005675    5725 generic.go:356] "Generic (PLEG): container finished" podID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerID="b5ef4a0601974940205e7fc5d1d56f76fd6d0beb5eada1775a7007342533f164" exitCode=0
Nov 01 12:27:43.005803 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:43.005761    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk" event={"ID":"03aea48c-4114-42da-b622-0bd5f8e7442f","Type":"ContainerDied","Data":"b5ef4a0601974940205e7fc5d1d56f76fd6d0beb5eada1775a7007342533f164"}
Nov 01 12:27:43.008972 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:43.008534    5725 generic.go:356] "Generic (PLEG): container finished" podID="790b64a7-9958-424d-b462-a29c3f97d8de" containerID="c879af3d2968222469227dc40af84c084e7edf3905495954ff79703d200e4219" exitCode=0
Nov 01 12:27:43.008972 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:43.008621    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2" event={"ID":"790b64a7-9958-424d-b462-a29c3f97d8de","Type":"ContainerDied","Data":"c879af3d2968222469227dc40af84c084e7edf3905495954ff79703d200e4219"}
Nov 01 12:27:44.018203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:44.018168    5725 generic.go:356] "Generic (PLEG): container finished" podID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerID="8851cec4cbdd931de352f7593737c8cd86124c11c4f9e732a3de1ad21a6060cb" exitCode=0
Nov 01 12:27:44.018581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:44.018370    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj" event={"ID":"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd","Type":"ContainerDied","Data":"8851cec4cbdd931de352f7593737c8cd86124c11c4f9e732a3de1ad21a6060cb"}
Nov 01 12:27:44.020493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:44.020304    5725 generic.go:356] "Generic (PLEG): container finished" podID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerID="864622b7d10cc948b9e94843cb086ae1bd4ab10c17c1014fc7e938117d0dd2b7" exitCode=0
Nov 01 12:27:44.020493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:44.020383    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t" event={"ID":"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5","Type":"ContainerDied","Data":"864622b7d10cc948b9e94843cb086ae1bd4ab10c17c1014fc7e938117d0dd2b7"}
Nov 01 12:27:44.024989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:44.024965    5725 generic.go:356] "Generic (PLEG): container finished" podID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerID="448ff112558b3617a10c17e3af3a7a0b631782ff8611278c22311cea3fafbbaf" exitCode=0
Nov 01 12:27:44.025103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:44.025038    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82" event={"ID":"ce09757e-992e-463d-a9bf-b6c807827d9d","Type":"ContainerDied","Data":"448ff112558b3617a10c17e3af3a7a0b631782ff8611278c22311cea3fafbbaf"}
Nov 01 12:27:44.026714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:44.026692    5725 generic.go:356] "Generic (PLEG): container finished" podID="f83f11a3-9457-4766-b72d-7988cda871cf" containerID="84618abe6bbcaf7099c9823a9d3999fa6efa35da8321f081ff5d35a9e76a9f16" exitCode=0
Nov 01 12:27:44.026815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:44.026729    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24" event={"ID":"f83f11a3-9457-4766-b72d-7988cda871cf","Type":"ContainerDied","Data":"84618abe6bbcaf7099c9823a9d3999fa6efa35da8321f081ff5d35a9e76a9f16"}
Nov 01 12:27:45.034972 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.034939    5725 generic.go:356] "Generic (PLEG): container finished" podID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerID="4cebcc1a11ef334c0ccca291c05072910fdc981da1c26a8b674e0faf1fa0128b" exitCode=0
Nov 01 12:27:45.035384 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.035029    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw" event={"ID":"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d","Type":"ContainerDied","Data":"4cebcc1a11ef334c0ccca291c05072910fdc981da1c26a8b674e0faf1fa0128b"}
Nov 01 12:27:45.037096 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.037070    5725 generic.go:356] "Generic (PLEG): container finished" podID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerID="4bd36e24e0cebb856a098254eb828f8d72e86ce12bbada9dac685ace1017a8cf" exitCode=0
Nov 01 12:27:45.037194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.037170    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw" event={"ID":"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba","Type":"ContainerDied","Data":"4bd36e24e0cebb856a098254eb828f8d72e86ce12bbada9dac685ace1017a8cf"}
Nov 01 12:27:45.042619 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.042598    5725 generic.go:356] "Generic (PLEG): container finished" podID="790b64a7-9958-424d-b462-a29c3f97d8de" containerID="106f3dde257a0be30d4327350a4af8ec2ba66ae1c56fec293db920e75b37b7e0" exitCode=0
Nov 01 12:27:45.042719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.042654    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2" event={"ID":"790b64a7-9958-424d-b462-a29c3f97d8de","Type":"ContainerDied","Data":"106f3dde257a0be30d4327350a4af8ec2ba66ae1c56fec293db920e75b37b7e0"}
Nov 01 12:27:45.045263 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.045235    5725 generic.go:356] "Generic (PLEG): container finished" podID="49703698-166d-4371-8556-da90a48de9d8" containerID="73f1043377d06b8cfd05c1798a58a0e3e9b33d1c3feea9ac5c0c03a0458b97dc" exitCode=0
Nov 01 12:27:45.045429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.045362    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk" event={"ID":"49703698-166d-4371-8556-da90a48de9d8","Type":"ContainerDied","Data":"73f1043377d06b8cfd05c1798a58a0e3e9b33d1c3feea9ac5c0c03a0458b97dc"}
Nov 01 12:27:45.051751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.051720    5725 generic.go:356] "Generic (PLEG): container finished" podID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerID="ff2678dbb8d5952d080533b1eb12cb69701345bdc7133010d7f3de2eb4db37da" exitCode=0
Nov 01 12:27:45.051864 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.051812    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2" event={"ID":"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca","Type":"ContainerDied","Data":"ff2678dbb8d5952d080533b1eb12cb69701345bdc7133010d7f3de2eb4db37da"}
Nov 01 12:27:45.054734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.054714    5725 generic.go:356] "Generic (PLEG): container finished" podID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerID="83a01eb3aa84393111a05c4f8135436ff2a34075cf97e58b7380a2b8016ea4f1" exitCode=0
Nov 01 12:27:45.054844 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.054795    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj" event={"ID":"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd","Type":"ContainerDied","Data":"83a01eb3aa84393111a05c4f8135436ff2a34075cf97e58b7380a2b8016ea4f1"}
Nov 01 12:27:45.057266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.057243    5725 generic.go:356] "Generic (PLEG): container finished" podID="757d9150-058a-4d63-b9e6-51359af63684" containerID="375d426ed16b0ad1765a0bfc887cc97bc4885295d3356f9d5f9e873152b5f411" exitCode=0
Nov 01 12:27:45.057377 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:45.057285    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6" event={"ID":"757d9150-058a-4d63-b9e6-51359af63684","Type":"ContainerDied","Data":"375d426ed16b0ad1765a0bfc887cc97bc4885295d3356f9d5f9e873152b5f411"}
Nov 01 12:27:46.083520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:46.083474    5725 generic.go:356] "Generic (PLEG): container finished" podID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerID="08fc2a930454744bad50056387c57e6fb49bb6266bafd8cd2e46be0f633624d9" exitCode=0
Nov 01 12:27:46.084020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:46.083678    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t" event={"ID":"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5","Type":"ContainerDied","Data":"08fc2a930454744bad50056387c57e6fb49bb6266bafd8cd2e46be0f633624d9"}
Nov 01 12:27:46.103521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:46.103396    5725 generic.go:356] "Generic (PLEG): container finished" podID="f83f11a3-9457-4766-b72d-7988cda871cf" containerID="f17d126bb197c610098f5a44b883c71ee535fce74dc806a884d21085c827ab89" exitCode=0
Nov 01 12:27:46.103521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:46.103486    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24" event={"ID":"f83f11a3-9457-4766-b72d-7988cda871cf","Type":"ContainerDied","Data":"f17d126bb197c610098f5a44b883c71ee535fce74dc806a884d21085c827ab89"}
Nov 01 12:27:46.112113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:46.112085    5725 generic.go:356] "Generic (PLEG): container finished" podID="90780cda-beee-4fe4-b767-0f28893febe4" containerID="0cb0f1dd8e2aa170fc93bfbdeac3eb3642d890939a509f3df994dacfdc488bd3" exitCode=0
Nov 01 12:27:46.112227 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:46.112152    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m" event={"ID":"90780cda-beee-4fe4-b767-0f28893febe4","Type":"ContainerDied","Data":"0cb0f1dd8e2aa170fc93bfbdeac3eb3642d890939a509f3df994dacfdc488bd3"}
Nov 01 12:27:48.621423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:48.619996    5725 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:27:52.290853 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:52.290805    5725 generic.go:356] "Generic (PLEG): container finished" podID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerID="37ad08f01ca7fa6703f598b45f7989109e475113f44811f11f1fb22aff059ba1" exitCode=0
Nov 01 12:27:52.291451 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:27:52.290856    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk" event={"ID":"03aea48c-4114-42da-b622-0bd5f8e7442f","Type":"ContainerDied","Data":"37ad08f01ca7fa6703f598b45f7989109e475113f44811f11f1fb22aff059ba1"}
Nov 01 12:28:00.850371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:00.849369    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:28:00.850371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:00.849496    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:28:00.850371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:00.849545    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:28:01.072552 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:28:01.072507    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:28:01.486117 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:01.485867    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" event={"ID":"9ec5523e-d2ad-414c-9186-003789159540","Type":"ContainerStarted","Data":"f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339"}
Nov 01 12:28:01.693410 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:01.693345    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" podStartSLOduration=1.6565593600000001 podStartE2EDuration="35.693306745s" totalImagesPullingTime="34.036747385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-01 12:28:01.6450175 +0000 UTC m=+323.152208776" watchObservedRunningTime="2026-11-01 12:28:01.693306745 +0000 UTC m=+323.200497975"
Nov 01 12:28:01.928407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:01.928348    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:28:01.928845 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:01.928468    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:28:01.928845 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:01.928522    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:28:02.360797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:02.360764    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:28:02.367270 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:02.367239    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:28:02.500492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:02.500449    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl" event={"ID":"de6b7627-2b0c-416b-b3fe-6f6d03e09089","Type":"ContainerStarted","Data":"658ca031c5d50a6a4b621f23a2c6b355a26023f7241c618fce562a59bd67fc83"}
Nov 01 12:28:02.500758 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:02.500525    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:28:02.502387 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:02.502359    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:28:02.614152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:02.614021    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl" podStartSLOduration=1.7646389359999999 podStartE2EDuration="36.61400081s" totalImagesPullingTime="34.849361874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-01 12:28:02.612801224 +0000 UTC m=+324.119992520" watchObservedRunningTime="2026-11-01 12:28:02.61400081 +0000 UTC m=+324.121192029"
Nov 01 12:28:02.653273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:02.653240    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:28:03.660374 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:03.660113    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:28:04.523839 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:04.523169    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:28:04.524253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:04.524225    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl"
Nov 01 12:28:10.626627 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:10.626578    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk" event={"ID":"03aea48c-4114-42da-b622-0bd5f8e7442f","Type":"ContainerStarted","Data":"76b96799210e29c12c36fcd358a4f8d25612ca4e8e8bd3394692c086b3baf78a"}
Nov 01 12:28:10.633834 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:10.633748    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82" event={"ID":"ce09757e-992e-463d-a9bf-b6c807827d9d","Type":"ContainerStarted","Data":"eb702616b86d4472500166b5c9be1e3e7b45fd92a540a6987c503f9194901711"}
Nov 01 12:28:10.859764 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:10.859650    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82" podStartSLOduration=3.5727306260000002 podStartE2EDuration="52.859634834s" totalImagesPullingTime="49.286904208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:27:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:10.858574778 +0000 UTC m=+332.365765997" watchObservedRunningTime="2026-11-01 12:28:10.859634834 +0000 UTC m=+332.366826050"
Nov 01 12:28:10.859954 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:10.859803    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk" podStartSLOduration=3.9983971670000003 podStartE2EDuration="50.859798156s" totalImagesPullingTime="46.861400989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:27:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:10.749371824 +0000 UTC m=+332.256563042" watchObservedRunningTime="2026-11-01 12:28:10.859798156 +0000 UTC m=+332.366989372"
Nov 01 12:28:11.645085 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:11.644514    5725 generic.go:356] "Generic (PLEG): container finished" podID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerID="76b96799210e29c12c36fcd358a4f8d25612ca4e8e8bd3394692c086b3baf78a" exitCode=0
Nov 01 12:28:11.645085 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:11.644647    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk" event={"ID":"03aea48c-4114-42da-b622-0bd5f8e7442f","Type":"ContainerDied","Data":"76b96799210e29c12c36fcd358a4f8d25612ca4e8e8bd3394692c086b3baf78a"}
Nov 01 12:28:11.652859 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:11.652604    5725 generic.go:356] "Generic (PLEG): container finished" podID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerID="eb702616b86d4472500166b5c9be1e3e7b45fd92a540a6987c503f9194901711" exitCode=0
Nov 01 12:28:11.652859 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:11.652670    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82" event={"ID":"ce09757e-992e-463d-a9bf-b6c807827d9d","Type":"ContainerDied","Data":"eb702616b86d4472500166b5c9be1e3e7b45fd92a540a6987c503f9194901711"}
Nov 01 12:28:12.659549 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.659516    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw" event={"ID":"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d","Type":"ContainerStarted","Data":"1ceb7da1149ae8efcb45c79d19b76f87383a2c6fd46af0955a1be523b3d6ace9"}
Nov 01 12:28:12.662400 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.662373    5725 generic.go:356] "Generic (PLEG): container finished" podID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerID="fd2d71f8487ac2077b44714d9e51efa8781976f818a195dbb4326adff20ee0e0" exitCode=0
Nov 01 12:28:12.662637 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.662619    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw" event={"ID":"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba","Type":"ContainerDied","Data":"fd2d71f8487ac2077b44714d9e51efa8781976f818a195dbb4326adff20ee0e0"}
Nov 01 12:28:12.666363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.666317    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2" event={"ID":"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca","Type":"ContainerStarted","Data":"a5fba4ed0ea476fd99c6084f64467c60fcbc56762d364e46793099caab8ac99e"}
Nov 01 12:28:12.767771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.767715    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw" podStartSLOduration=3.419000788 podStartE2EDuration="47.767701273s" totalImagesPullingTime="44.348700485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:27:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:12.767121692 +0000 UTC m=+334.274312908" watchObservedRunningTime="2026-11-01 12:28:12.767701273 +0000 UTC m=+334.274892486"
Nov 01 12:28:12.810810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.810781    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:28:12.893442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.893411    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:28:12.967307 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.967268    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-util\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-util\") pod \"03aea48c-4114-42da-b622-0bd5f8e7442f\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") "
Nov 01 12:28:12.967431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.967353    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-bundle\") pod \"03aea48c-4114-42da-b622-0bd5f8e7442f\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") "
Nov 01 12:28:12.967431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.967392    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03aea48c-4114-42da-b622-0bd5f8e7442f-kube-api-access-crrw8\" (UniqueName: \"kubernetes.io/projected/03aea48c-4114-42da-b622-0bd5f8e7442f-kube-api-access-crrw8\") pod \"03aea48c-4114-42da-b622-0bd5f8e7442f\" (UID: \"03aea48c-4114-42da-b622-0bd5f8e7442f\") "
Nov 01 12:28:12.969075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.968101    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-bundle" pod "03aea48c-4114-42da-b622-0bd5f8e7442f" (UID: "03aea48c-4114-42da-b622-0bd5f8e7442f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:12.970490 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.970445    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03aea48c-4114-42da-b622-0bd5f8e7442f-kube-api-access-crrw8" pod "03aea48c-4114-42da-b622-0bd5f8e7442f" (UID: "03aea48c-4114-42da-b622-0bd5f8e7442f"). InnerVolumeSpecName "kube-api-access-crrw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:12.973148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.973123    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-util" pod "03aea48c-4114-42da-b622-0bd5f8e7442f" (UID: "03aea48c-4114-42da-b622-0bd5f8e7442f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:12.983919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:12.983874    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2" podStartSLOduration=3.271391709 podStartE2EDuration="45.983859004s" totalImagesPullingTime="42.712467295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:27:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:12.982187094 +0000 UTC m=+334.489378310" watchObservedRunningTime="2026-11-01 12:28:12.983859004 +0000 UTC m=+334.491050215"
Nov 01 12:28:13.068775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.068741    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce09757e-992e-463d-a9bf-b6c807827d9d-kube-api-access-5wz89\" (UniqueName: \"kubernetes.io/projected/ce09757e-992e-463d-a9bf-b6c807827d9d-kube-api-access-5wz89\") pod \"ce09757e-992e-463d-a9bf-b6c807827d9d\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") "
Nov 01 12:28:13.068936 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.068816    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-util\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-util\") pod \"ce09757e-992e-463d-a9bf-b6c807827d9d\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") "
Nov 01 12:28:13.068936 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.068863    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-bundle\") pod \"ce09757e-992e-463d-a9bf-b6c807827d9d\" (UID: \"ce09757e-992e-463d-a9bf-b6c807827d9d\") "
Nov 01 12:28:13.069257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.069237    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:13.069327 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.069263    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03aea48c-4114-42da-b622-0bd5f8e7442f-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:13.069327 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.069279    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crrw8\" (UniqueName: \"kubernetes.io/projected/03aea48c-4114-42da-b622-0bd5f8e7442f-kube-api-access-crrw8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:13.069598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.069579    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-bundle" pod "ce09757e-992e-463d-a9bf-b6c807827d9d" (UID: "ce09757e-992e-463d-a9bf-b6c807827d9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:13.071287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.071260    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce09757e-992e-463d-a9bf-b6c807827d9d-kube-api-access-5wz89" pod "ce09757e-992e-463d-a9bf-b6c807827d9d" (UID: "ce09757e-992e-463d-a9bf-b6c807827d9d"). InnerVolumeSpecName "kube-api-access-5wz89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:13.075028 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.075002    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-util" pod "ce09757e-992e-463d-a9bf-b6c807827d9d" (UID: "ce09757e-992e-463d-a9bf-b6c807827d9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:13.170235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.170163    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wz89\" (UniqueName: \"kubernetes.io/projected/ce09757e-992e-463d-a9bf-b6c807827d9d-kube-api-access-5wz89\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:13.170235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.170200    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:13.170235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.170215    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce09757e-992e-463d-a9bf-b6c807827d9d-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:13.672725 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.672657    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:28:13.673289 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.673266    5725 generic.go:356] "Generic (PLEG): container finished" podID="90780cda-beee-4fe4-b767-0f28893febe4" containerID="05c940b7d3dc54a97b48333bbb232615aee66751c6bcd1baee387a9e4c90a585" exitCode=1
Nov 01 12:28:13.673368 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.673350    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m" event={"ID":"90780cda-beee-4fe4-b767-0f28893febe4","Type":"ContainerDied","Data":"05c940b7d3dc54a97b48333bbb232615aee66751c6bcd1baee387a9e4c90a585"}
Nov 01 12:28:13.675500 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.675474    5725 generic.go:356] "Generic (PLEG): container finished" podID="757d9150-058a-4d63-b9e6-51359af63684" containerID="2f0bea85634b2d0af14d80e5a2a9a21ce86624171085e94671190923d4f5fe76" exitCode=0
Nov 01 12:28:13.675627 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.675566    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6" event={"ID":"757d9150-058a-4d63-b9e6-51359af63684","Type":"ContainerDied","Data":"2f0bea85634b2d0af14d80e5a2a9a21ce86624171085e94671190923d4f5fe76"}
Nov 01 12:28:13.677556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.677537    5725 generic.go:356] "Generic (PLEG): container finished" podID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerID="1ceb7da1149ae8efcb45c79d19b76f87383a2c6fd46af0955a1be523b3d6ace9" exitCode=0
Nov 01 12:28:13.677656 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.677617    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw" event={"ID":"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d","Type":"ContainerDied","Data":"1ceb7da1149ae8efcb45c79d19b76f87383a2c6fd46af0955a1be523b3d6ace9"}
Nov 01 12:28:13.679352 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.679330    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk" event={"ID":"03aea48c-4114-42da-b622-0bd5f8e7442f","Type":"ContainerDied","Data":"0b770546b388db9830c9d4002c2fdd619e5b3bde86548c850bf91f79ade175e8"}
Nov 01 12:28:13.679472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.679371    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b770546b388db9830c9d4002c2fdd619e5b3bde86548c850bf91f79ade175e8"
Nov 01 12:28:13.679472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.679336    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64484zdk"
Nov 01 12:28:13.681499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.681473    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t" event={"ID":"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5","Type":"ContainerStarted","Data":"6a24f300b0c6691ab2bccfc75e15b7ad1b4a9034c175fcb2b6050340af0d33a7"}
Nov 01 12:28:13.683588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.683562    5725 generic.go:356] "Generic (PLEG): container finished" podID="790b64a7-9958-424d-b462-a29c3f97d8de" containerID="17830fe1db90cb9681a7fa851e01aca142de7c13a037b933f04c21e7b00c0a16" exitCode=0
Nov 01 12:28:13.683676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.683634    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2" event={"ID":"790b64a7-9958-424d-b462-a29c3f97d8de","Type":"ContainerDied","Data":"17830fe1db90cb9681a7fa851e01aca142de7c13a037b933f04c21e7b00c0a16"}
Nov 01 12:28:13.685701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.685678    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk" event={"ID":"49703698-166d-4371-8556-da90a48de9d8","Type":"ContainerStarted","Data":"7db5b78dce6016936fc5093792a95d8f723958a6db6bcb05614902203a297633"}
Nov 01 12:28:13.687477 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.687461    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82"
Nov 01 12:28:13.687585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.687471    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmhk82" event={"ID":"ce09757e-992e-463d-a9bf-b6c807827d9d","Type":"ContainerDied","Data":"bb91edb4ecdb3a65ff6556bac71eef3ce63ed135a604a61edece93a65d150db1"}
Nov 01 12:28:13.687585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.687509    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb91edb4ecdb3a65ff6556bac71eef3ce63ed135a604a61edece93a65d150db1"
Nov 01 12:28:13.689984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.689965    5725 generic.go:356] "Generic (PLEG): container finished" podID="f83f11a3-9457-4766-b72d-7988cda871cf" containerID="485b97791fa9daa1001ef059156446e42c67f3f886c4b0352a724127d9c64dab" exitCode=0
Nov 01 12:28:13.690100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.690033    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24" event={"ID":"f83f11a3-9457-4766-b72d-7988cda871cf","Type":"ContainerDied","Data":"485b97791fa9daa1001ef059156446e42c67f3f886c4b0352a724127d9c64dab"}
Nov 01 12:28:13.692094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.692072    5725 generic.go:356] "Generic (PLEG): container finished" podID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerID="a5fba4ed0ea476fd99c6084f64467c60fcbc56762d364e46793099caab8ac99e" exitCode=0
Nov 01 12:28:13.692200 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.692130    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2" event={"ID":"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca","Type":"ContainerDied","Data":"a5fba4ed0ea476fd99c6084f64467c60fcbc56762d364e46793099caab8ac99e"}
Nov 01 12:28:13.694183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.694163    5725 generic.go:356] "Generic (PLEG): container finished" podID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerID="b992e0427fb95c5be8acf5e492a3558b61ae17016f2902734060ddc1d7227a5f" exitCode=0
Nov 01 12:28:13.694273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.694228    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj" event={"ID":"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd","Type":"ContainerDied","Data":"b992e0427fb95c5be8acf5e492a3558b61ae17016f2902734060ddc1d7227a5f"}
Nov 01 12:28:13.842450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.842420    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:28:13.896267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.896212    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t" podStartSLOduration=3.032011132 podStartE2EDuration="50.89619638s" totalImagesPullingTime="47.864185248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:27:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:13.89572528 +0000 UTC m=+335.402916497" watchObservedRunningTime="2026-11-01 12:28:13.89619638 +0000 UTC m=+335.403387596"
Nov 01 12:28:13.976449 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.976421    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-util\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-util\") pod \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") "
Nov 01 12:28:13.976610 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.976477    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-bundle\") pod \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") "
Nov 01 12:28:13.976610 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.976512    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-kube-api-access-59t7p\" (UniqueName: \"kubernetes.io/projected/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-kube-api-access-59t7p\") pod \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\" (UID: \"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba\") "
Nov 01 12:28:13.976949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.976918    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-bundle" pod "9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" (UID: "9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:13.978544 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.978526    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-kube-api-access-59t7p" pod "9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" (UID: "9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba"). InnerVolumeSpecName "kube-api-access-59t7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:13.982684 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:13.982662    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-util" pod "9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" (UID: "9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:14.078087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.078031    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:14.078087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.078081    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:14.078087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.078092    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59t7p\" (UniqueName: \"kubernetes.io/projected/9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba-kube-api-access-59t7p\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:14.646287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.646234    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk" podStartSLOduration=3.901205444 podStartE2EDuration="54.646219371s" totalImagesPullingTime="50.745013927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:27:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:14.645651985 +0000 UTC m=+336.152843202" watchObservedRunningTime="2026-11-01 12:28:14.646219371 +0000 UTC m=+336.153410588"
Nov 01 12:28:14.700822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.700789    5725 generic.go:356] "Generic (PLEG): container finished" podID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerID="6a24f300b0c6691ab2bccfc75e15b7ad1b4a9034c175fcb2b6050340af0d33a7" exitCode=0
Nov 01 12:28:14.701270 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.700875    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t" event={"ID":"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5","Type":"ContainerDied","Data":"6a24f300b0c6691ab2bccfc75e15b7ad1b4a9034c175fcb2b6050340af0d33a7"}
Nov 01 12:28:14.702833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.702813    5725 generic.go:356] "Generic (PLEG): container finished" podID="49703698-166d-4371-8556-da90a48de9d8" containerID="7db5b78dce6016936fc5093792a95d8f723958a6db6bcb05614902203a297633" exitCode=0
Nov 01 12:28:14.702911 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.702892    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk" event={"ID":"49703698-166d-4371-8556-da90a48de9d8","Type":"ContainerDied","Data":"7db5b78dce6016936fc5093792a95d8f723958a6db6bcb05614902203a297633"}
Nov 01 12:28:14.704571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.704545    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw" event={"ID":"9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba","Type":"ContainerDied","Data":"a86d11e25f369c14181ad428ad0d3b999b98c9a07c12694a6ec966b419d63366"}
Nov 01 12:28:14.704571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.704568    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kv7qw"
Nov 01 12:28:14.704711 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.704588    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a86d11e25f369c14181ad428ad0d3b999b98c9a07c12694a6ec966b419d63366"
Nov 01 12:28:14.889698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.889673    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:28:14.980963 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.980943    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:28:14.985144 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.985125    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83f11a3-9457-4766-b72d-7988cda871cf-kube-api-access-8tm79\" (UniqueName: \"kubernetes.io/projected/f83f11a3-9457-4766-b72d-7988cda871cf-kube-api-access-8tm79\") pod \"f83f11a3-9457-4766-b72d-7988cda871cf\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") "
Nov 01 12:28:14.985257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.985204    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-bundle\") pod \"f83f11a3-9457-4766-b72d-7988cda871cf\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") "
Nov 01 12:28:14.985378 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.985293    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-util\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-util\") pod \"f83f11a3-9457-4766-b72d-7988cda871cf\" (UID: \"f83f11a3-9457-4766-b72d-7988cda871cf\") "
Nov 01 12:28:14.985717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.985692    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-bundle" pod "f83f11a3-9457-4766-b72d-7988cda871cf" (UID: "f83f11a3-9457-4766-b72d-7988cda871cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:14.985899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.985873    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:28:14.987734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.987711    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83f11a3-9457-4766-b72d-7988cda871cf-kube-api-access-8tm79" pod "f83f11a3-9457-4766-b72d-7988cda871cf" (UID: "f83f11a3-9457-4766-b72d-7988cda871cf"). InnerVolumeSpecName "kube-api-access-8tm79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:14.989471 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.989450    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:28:14.992304 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.992278    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-util" pod "f83f11a3-9457-4766-b72d-7988cda871cf" (UID: "f83f11a3-9457-4766-b72d-7988cda871cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:14.994027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.994006    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:28:14.996094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.996074    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:28:14.997634 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:14.997616    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:28:15.001004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.000987    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:28:15.086136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086100    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-kube-api-access-klrts\" (UniqueName: \"kubernetes.io/projected/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-kube-api-access-klrts\") pod \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") "
Nov 01 12:28:15.086296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086153    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-bundle\") pod \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") "
Nov 01 12:28:15.086296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086190    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-util\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-util\") pod \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") "
Nov 01 12:28:15.086296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086221    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-bundle\") pod \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") "
Nov 01 12:28:15.086296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086273    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-kube-api-access-xzkbb\" (UniqueName: \"kubernetes.io/projected/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-kube-api-access-xzkbb\") pod \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\" (UID: \"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d\") "
Nov 01 12:28:15.086525 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086333    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-bundle\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-bundle\") pod \"757d9150-058a-4d63-b9e6-51359af63684\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") "
Nov 01 12:28:15.086525 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086361    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757d9150-058a-4d63-b9e6-51359af63684-kube-api-access-lfhcp\" (UniqueName: \"kubernetes.io/projected/757d9150-058a-4d63-b9e6-51359af63684-kube-api-access-lfhcp\") pod \"757d9150-058a-4d63-b9e6-51359af63684\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") "
Nov 01 12:28:15.086525 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086402    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-util\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-util\") pod \"757d9150-058a-4d63-b9e6-51359af63684\" (UID: \"757d9150-058a-4d63-b9e6-51359af63684\") "
Nov 01 12:28:15.086525 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086426    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-util\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-util\") pod \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\" (UID: \"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd\") "
Nov 01 12:28:15.086720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086669    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.086720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086690    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tm79\" (UniqueName: \"kubernetes.io/projected/f83f11a3-9457-4766-b72d-7988cda871cf-kube-api-access-8tm79\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.086720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.086705    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f83f11a3-9457-4766-b72d-7988cda871cf-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.087404 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.087372    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-bundle" pod "57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" (UID: "57d1c1b7-fdf1-42c4-996f-be697a1a3cdd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.087504 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.087467    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-bundle" pod "3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" (UID: "3ff279e1-dbdd-4fa3-a9c2-70ab2971870d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.087588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.087564    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-bundle" pod "757d9150-058a-4d63-b9e6-51359af63684" (UID: "757d9150-058a-4d63-b9e6-51359af63684"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.088849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.088820    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-kube-api-access-klrts" pod "57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" (UID: "57d1c1b7-fdf1-42c4-996f-be697a1a3cdd"). InnerVolumeSpecName "kube-api-access-klrts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:15.089097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.089078    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757d9150-058a-4d63-b9e6-51359af63684-kube-api-access-lfhcp" pod "757d9150-058a-4d63-b9e6-51359af63684" (UID: "757d9150-058a-4d63-b9e6-51359af63684"). InnerVolumeSpecName "kube-api-access-lfhcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:15.089222 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.089199    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-kube-api-access-xzkbb" pod "3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" (UID: "3ff279e1-dbdd-4fa3-a9c2-70ab2971870d"). InnerVolumeSpecName "kube-api-access-xzkbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:15.093979 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.093775    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-util" pod "3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" (UID: "3ff279e1-dbdd-4fa3-a9c2-70ab2971870d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.095354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.095329    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-util" pod "757d9150-058a-4d63-b9e6-51359af63684" (UID: "757d9150-058a-4d63-b9e6-51359af63684"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.095681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.095663    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-util" pod "57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" (UID: "57d1c1b7-fdf1-42c4-996f-be697a1a3cdd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.187338 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187242    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-util\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-util\") pod \"90780cda-beee-4fe4-b767-0f28893febe4\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") "
Nov 01 12:28:15.187338 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187287    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-bundle\") pod \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") "
Nov 01 12:28:15.187338 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187318    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-util\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-util\") pod \"790b64a7-9958-424d-b462-a29c3f97d8de\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") "
Nov 01 12:28:15.187338 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187334    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-bundle\") pod \"90780cda-beee-4fe4-b767-0f28893febe4\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") "
Nov 01 12:28:15.187673 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187351    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-util\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-util\") pod \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") "
Nov 01 12:28:15.187673 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187393    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-bundle\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-bundle\") pod \"790b64a7-9958-424d-b462-a29c3f97d8de\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") "
Nov 01 12:28:15.187673 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187425    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790b64a7-9958-424d-b462-a29c3f97d8de-kube-api-access-5tgsh\" (UniqueName: \"kubernetes.io/projected/790b64a7-9958-424d-b462-a29c3f97d8de-kube-api-access-5tgsh\") pod \"790b64a7-9958-424d-b462-a29c3f97d8de\" (UID: \"790b64a7-9958-424d-b462-a29c3f97d8de\") "
Nov 01 12:28:15.187673 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187472    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90780cda-beee-4fe4-b767-0f28893febe4-kube-api-access-jmvw7\" (UniqueName: \"kubernetes.io/projected/90780cda-beee-4fe4-b767-0f28893febe4-kube-api-access-jmvw7\") pod \"90780cda-beee-4fe4-b767-0f28893febe4\" (UID: \"90780cda-beee-4fe4-b767-0f28893febe4\") "
Nov 01 12:28:15.187673 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187538    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-kube-api-access-zz8qq\" (UniqueName: \"kubernetes.io/projected/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-kube-api-access-zz8qq\") pod \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\" (UID: \"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca\") "
Nov 01 12:28:15.187903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187780    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzkbb\" (UniqueName: \"kubernetes.io/projected/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-kube-api-access-xzkbb\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.187903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187800    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.187903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187816    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhcp\" (UniqueName: \"kubernetes.io/projected/757d9150-058a-4d63-b9e6-51359af63684-kube-api-access-lfhcp\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.187903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187829    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/757d9150-058a-4d63-b9e6-51359af63684-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.187903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187842    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.187903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187855    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klrts\" (UniqueName: \"kubernetes.io/projected/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-kube-api-access-klrts\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.187903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187869    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d1c1b7-fdf1-42c4-996f-be697a1a3cdd-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.187903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187882    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.187903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187895    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff279e1-dbdd-4fa3-a9c2-70ab2971870d-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.188369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187953    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-bundle" pod "790b64a7-9958-424d-b462-a29c3f97d8de" (UID: "790b64a7-9958-424d-b462-a29c3f97d8de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.188369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187963    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-bundle" pod "90780cda-beee-4fe4-b767-0f28893febe4" (UID: "90780cda-beee-4fe4-b767-0f28893febe4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.188369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.187997    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-bundle" pod "45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" (UID: "45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.190659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.190037    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90780cda-beee-4fe4-b767-0f28893febe4-kube-api-access-jmvw7" pod "90780cda-beee-4fe4-b767-0f28893febe4" (UID: "90780cda-beee-4fe4-b767-0f28893febe4"). InnerVolumeSpecName "kube-api-access-jmvw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:15.190659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.190611    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-kube-api-access-zz8qq" pod "45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" (UID: "45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca"). InnerVolumeSpecName "kube-api-access-zz8qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:15.190839 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.190710    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790b64a7-9958-424d-b462-a29c3f97d8de-kube-api-access-5tgsh" pod "790b64a7-9958-424d-b462-a29c3f97d8de" (UID: "790b64a7-9958-424d-b462-a29c3f97d8de"). InnerVolumeSpecName "kube-api-access-5tgsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:15.195920 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.195885    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-util" pod "45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" (UID: "45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.196389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.196356    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-util" pod "790b64a7-9958-424d-b462-a29c3f97d8de" (UID: "790b64a7-9958-424d-b462-a29c3f97d8de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.196824 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.196795    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-util" pod "90780cda-beee-4fe4-b767-0f28893febe4" (UID: "90780cda-beee-4fe4-b767-0f28893febe4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.289162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.289124    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz8qq\" (UniqueName: \"kubernetes.io/projected/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-kube-api-access-zz8qq\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.289162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.289153    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.289162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.289163    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.289162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.289171    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.289416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.289179    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90780cda-beee-4fe4-b767-0f28893febe4-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.289416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.289187    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.289416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.289194    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/790b64a7-9958-424d-b462-a29c3f97d8de-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.289416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.289202    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tgsh\" (UniqueName: \"kubernetes.io/projected/790b64a7-9958-424d-b462-a29c3f97d8de-kube-api-access-5tgsh\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.289416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.289210    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvw7\" (UniqueName: \"kubernetes.io/projected/90780cda-beee-4fe4-b767-0f28893febe4-kube-api-access-jmvw7\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:15.712267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.712237    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24"
Nov 01 12:28:15.712752 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.712235    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799j24" event={"ID":"f83f11a3-9457-4766-b72d-7988cda871cf","Type":"ContainerDied","Data":"4fae626bbc44295a1ac7fb7b93682e7f71f6e4bd86dd34136fd70a8f3833272d"}
Nov 01 12:28:15.712752 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.712366    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fae626bbc44295a1ac7fb7b93682e7f71f6e4bd86dd34136fd70a8f3833272d"
Nov 01 12:28:15.714320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.714289    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2" event={"ID":"45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca","Type":"ContainerDied","Data":"768c1fcb4a4aca04768baa3ebee4089dca293668070ca720282eca061f2b92c9"}
Nov 01 12:28:15.714320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.714317    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvldk2"
Nov 01 12:28:15.714483 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.714330    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="768c1fcb4a4aca04768baa3ebee4089dca293668070ca720282eca061f2b92c9"
Nov 01 12:28:15.716351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.716332    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj"
Nov 01 12:28:15.716351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.716332    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axrxrj" event={"ID":"57d1c1b7-fdf1-42c4-996f-be697a1a3cdd","Type":"ContainerDied","Data":"95995ad1aa51d90c124023cab4a20b0c59e5c6da5b6cb9c2d166bead18ae04a2"}
Nov 01 12:28:15.716535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.716376    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95995ad1aa51d90c124023cab4a20b0c59e5c6da5b6cb9c2d166bead18ae04a2"
Nov 01 12:28:15.717780 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.717760    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:28:15.718568 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.718545    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m"
Nov 01 12:28:15.718658 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.718545    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m" event={"ID":"90780cda-beee-4fe4-b767-0f28893febe4","Type":"ContainerDied","Data":"0217ec8a7a4e1691e9518d8e6d3b3e460b9add0a18883bd4537fafd4e25a9369"}
Nov 01 12:28:15.718737 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.718675    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0217ec8a7a4e1691e9518d8e6d3b3e460b9add0a18883bd4537fafd4e25a9369"
Nov 01 12:28:15.720709 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.720677    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6"
Nov 01 12:28:15.720709 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.720691    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwkzc6" event={"ID":"757d9150-058a-4d63-b9e6-51359af63684","Type":"ContainerDied","Data":"0897f8b7e75948982186bd40d8b19f6fb7aab4adc44217768191772ed7b401af"}
Nov 01 12:28:15.720867 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.720733    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0897f8b7e75948982186bd40d8b19f6fb7aab4adc44217768191772ed7b401af"
Nov 01 12:28:15.723584 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.723495    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw"
Nov 01 12:28:15.723584 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.723499    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbmlvw" event={"ID":"3ff279e1-dbdd-4fa3-a9c2-70ab2971870d","Type":"ContainerDied","Data":"aada9f758e60ba02286592c3b054c7b62d027c139349e4bffe43e0c4dd8fab21"}
Nov 01 12:28:15.723733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.723609    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aada9f758e60ba02286592c3b054c7b62d027c139349e4bffe43e0c4dd8fab21"
Nov 01 12:28:15.726615 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.726590    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2" event={"ID":"790b64a7-9958-424d-b462-a29c3f97d8de","Type":"ContainerDied","Data":"c601c801746ade78a0e62ee34866bde92b484b1a8436cea80f05038b3942c662"}
Nov 01 12:28:15.726708 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.726633    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c601c801746ade78a0e62ee34866bde92b484b1a8436cea80f05038b3942c662"
Nov 01 12:28:15.726848 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.726831    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65f9x2"
Nov 01 12:28:15.874523 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.874496    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:28:15.881216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.881194    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:28:15.995497 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.995458    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-util\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-util\") pod \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") "
Nov 01 12:28:15.995693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.995536    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-util\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-util\") pod \"49703698-166d-4371-8556-da90a48de9d8\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") "
Nov 01 12:28:15.995693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.995559    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-bundle\") pod \"49703698-166d-4371-8556-da90a48de9d8\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") "
Nov 01 12:28:15.995693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.995578    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-kube-api-access-cn269\" (UniqueName: \"kubernetes.io/projected/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-kube-api-access-cn269\") pod \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") "
Nov 01 12:28:15.995693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.995607    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-bundle\") pod \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\" (UID: \"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5\") "
Nov 01 12:28:15.995693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.995632    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49703698-166d-4371-8556-da90a48de9d8-kube-api-access-6m8j6\" (UniqueName: \"kubernetes.io/projected/49703698-166d-4371-8556-da90a48de9d8-kube-api-access-6m8j6\") pod \"49703698-166d-4371-8556-da90a48de9d8\" (UID: \"49703698-166d-4371-8556-da90a48de9d8\") "
Nov 01 12:28:15.996341 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.996300    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-bundle" pod "3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" (UID: "3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.996498 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.996376    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-bundle" pod "49703698-166d-4371-8556-da90a48de9d8" (UID: "49703698-166d-4371-8556-da90a48de9d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:15.997942 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.997921    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49703698-166d-4371-8556-da90a48de9d8-kube-api-access-6m8j6" pod "49703698-166d-4371-8556-da90a48de9d8" (UID: "49703698-166d-4371-8556-da90a48de9d8"). InnerVolumeSpecName "kube-api-access-6m8j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:15.998086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:15.998069    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-kube-api-access-cn269" pod "3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" (UID: "3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5"). InnerVolumeSpecName "kube-api-access-cn269". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:28:16.001810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.001790    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-util" pod "49703698-166d-4371-8556-da90a48de9d8" (UID: "49703698-166d-4371-8556-da90a48de9d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:16.003158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.003133    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-util" pod "3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" (UID: "3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:28:16.096870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.096836    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:16.096870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.096865    5725 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-util\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:16.096870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.096876    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49703698-166d-4371-8556-da90a48de9d8-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:16.097148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.096885    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn269\" (UniqueName: \"kubernetes.io/projected/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-kube-api-access-cn269\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:16.097148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.096894    5725 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:16.097148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.096903    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m8j6\" (UniqueName: \"kubernetes.io/projected/49703698-166d-4371-8556-da90a48de9d8-kube-api-access-6m8j6\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:28:16.738597 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.738561    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t" event={"ID":"3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5","Type":"ContainerDied","Data":"3fee40f7d4224c548029592bfc0fa5f8c5e5b972c91568c80b588a600c28ada7"}
Nov 01 12:28:16.739007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.738644    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fee40f7d4224c548029592bfc0fa5f8c5e5b972c91568c80b588a600c28ada7"
Nov 01 12:28:16.739007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.738574    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71nnh8t"
Nov 01 12:28:16.740725 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.740685    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk" event={"ID":"49703698-166d-4371-8556-da90a48de9d8","Type":"ContainerDied","Data":"d3baa21bdc6474ee4e36448e7d881795a5eb20eabfc2f7c8969b3ecf79b0b158"}
Nov 01 12:28:16.740725 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.740707    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ag4glk"
Nov 01 12:28:16.740927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:16.740737    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3baa21bdc6474ee4e36448e7d881795a5eb20eabfc2f7c8969b3ecf79b0b158"
Nov 01 12:28:30.647229 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647194    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"]
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647493    5725 cpu_manager.go:414] "containerMap: removing container" podUID="757d9150-058a-4d63-b9e6-51359af63684" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647503    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757d9150-058a-4d63-b9e6-51359af63684" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647512    5725 cpu_manager.go:414] "containerMap: removing container" podUID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647517    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647524    5725 cpu_manager.go:414] "containerMap: removing container" podUID="90780cda-beee-4fe4-b767-0f28893febe4" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647529    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90780cda-beee-4fe4-b767-0f28893febe4" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647541    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647546    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647552    5725 cpu_manager.go:414] "containerMap: removing container" podUID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerName="util"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647556    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerName="util"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647563    5725 cpu_manager.go:414] "containerMap: removing container" podUID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerName="util"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647567    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerName="util"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647574    5725 cpu_manager.go:414] "containerMap: removing container" podUID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647578    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerName="pull"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647586    5725 cpu_manager.go:414] "containerMap: removing container" podUID="49703698-166d-4371-8556-da90a48de9d8" containerName="util"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647590    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49703698-166d-4371-8556-da90a48de9d8" containerName="util"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647598    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f83f11a3-9457-4766-b72d-7988cda871cf" containerName="extract"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647602    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83f11a3-9457-4766-b72d-7988cda871cf" containerName="extract"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647611    5725 cpu_manager.go:414] "containerMap: removing container" podUID="790b64a7-9958-424d-b462-a29c3f97d8de" containerName="extract"
Nov 01 12:28:30.647612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647616    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790b64a7-9958-424d-b462-a29c3f97d8de" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647625    5725 cpu_manager.go:414] "containerMap: removing container" podUID="49703698-166d-4371-8556-da90a48de9d8" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647629    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49703698-166d-4371-8556-da90a48de9d8" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647636    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647640    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647646    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f83f11a3-9457-4766-b72d-7988cda871cf" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647650    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83f11a3-9457-4766-b72d-7988cda871cf" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647656    5725 cpu_manager.go:414] "containerMap: removing container" podUID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647660    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647668    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647672    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647678    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647682    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647690    5725 cpu_manager.go:414] "containerMap: removing container" podUID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647694    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647700    5725 cpu_manager.go:414] "containerMap: removing container" podUID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647703    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647710    5725 cpu_manager.go:414] "containerMap: removing container" podUID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647715    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647722    5725 cpu_manager.go:414] "containerMap: removing container" podUID="90780cda-beee-4fe4-b767-0f28893febe4" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647725    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90780cda-beee-4fe4-b767-0f28893febe4" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647732    5725 cpu_manager.go:414] "containerMap: removing container" podUID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647736    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647742    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f83f11a3-9457-4766-b72d-7988cda871cf" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647746    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83f11a3-9457-4766-b72d-7988cda871cf" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647759    5725 cpu_manager.go:414] "containerMap: removing container" podUID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647764    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647770    5725 cpu_manager.go:414] "containerMap: removing container" podUID="49703698-166d-4371-8556-da90a48de9d8" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647774    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49703698-166d-4371-8556-da90a48de9d8" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647779    5725 cpu_manager.go:414] "containerMap: removing container" podUID="757d9150-058a-4d63-b9e6-51359af63684" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647783    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757d9150-058a-4d63-b9e6-51359af63684" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647792    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647797    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647803    5725 cpu_manager.go:414] "containerMap: removing container" podUID="757d9150-058a-4d63-b9e6-51359af63684" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647808    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757d9150-058a-4d63-b9e6-51359af63684" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647814    5725 cpu_manager.go:414] "containerMap: removing container" podUID="790b64a7-9958-424d-b462-a29c3f97d8de" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647818    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790b64a7-9958-424d-b462-a29c3f97d8de" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647824    5725 cpu_manager.go:414] "containerMap: removing container" podUID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647828    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647833    5725 cpu_manager.go:414] "containerMap: removing container" podUID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647837    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647843    5725 cpu_manager.go:414] "containerMap: removing container" podUID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647848    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647854    5725 cpu_manager.go:414] "containerMap: removing container" podUID="90780cda-beee-4fe4-b767-0f28893febe4" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647857    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90780cda-beee-4fe4-b767-0f28893febe4" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647863    5725 cpu_manager.go:414] "containerMap: removing container" podUID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647866    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647873    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647879    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerName="util"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647884    5725 cpu_manager.go:414] "containerMap: removing container" podUID="790b64a7-9958-424d-b462-a29c3f97d8de" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647888    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790b64a7-9958-424d-b462-a29c3f97d8de" containerName="pull"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647894    5725 cpu_manager.go:414] "containerMap: removing container" podUID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647898    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647942    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff279e1-dbdd-4fa3-a9c2-70ab2971870d" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647949    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3ac5ee-f5a4-4bc4-84fc-f7d1e1a52eb5" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647955    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="03aea48c-4114-42da-b622-0bd5f8e7442f" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647960    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d1c1b7-fdf1-42c4-996f-be697a1a3cdd" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647967    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eedbaf1-cd65-4c1a-9e38-99fb7fc36aba" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647972    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce09757e-992e-463d-a9bf-b6c807827d9d" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647978    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="49703698-166d-4371-8556-da90a48de9d8" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647983    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="757d9150-058a-4d63-b9e6-51359af63684" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647988    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="90780cda-beee-4fe4-b767-0f28893febe4" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647993    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f0a0c8-7e3b-45b1-8c0c-4b73676f1cca" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.647999    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83f11a3-9457-4766-b72d-7988cda871cf" containerName="extract"
Nov 01 12:28:30.648191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.648004    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="790b64a7-9958-424d-b462-a29c3f97d8de" containerName="extract"
Nov 01 12:28:30.651179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.651163    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.700739 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.700705    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 12:28:30.700937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.700760    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 12:28:30.700937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.700776    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 12:28:30.700937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.700802    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 12:28:30.700937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.700878    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hqxjc\""
Nov 01 12:28:30.700937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.700920    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 12:28:30.707567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.707546    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.707624 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.707580    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.707624 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.707603    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-metrics-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.707624 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.707620    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-manifests\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.707743 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.707724    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krvlm\" (UniqueName: \"kubernetes.io/projected/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-kube-api-access-krvlm\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.707800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.707786    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-tmp\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.755046 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.755008    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"]
Nov 01 12:28:30.808461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.808431    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-tmp\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.808645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.808474    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.808645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.808497    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.808645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.808586    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-metrics-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.808645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.808634    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-manifests\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.808831 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.808669    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krvlm\" (UniqueName: \"kubernetes.io/projected/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-kube-api-access-krvlm\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.809104 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.809081    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-manifests\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.810849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.810822    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-tmp\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.811094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.811071    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.811290 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.811271    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.811362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.811351    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-metrics-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.866127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.866092    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krvlm\" (UniqueName: \"kubernetes.io/projected/1a9bd825-433d-4e5f-9bd5-326e83cea7fb-kube-api-access-krvlm\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs\" (UID: \"1a9bd825-433d-4e5f-9bd5-326e83cea7fb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:30.965515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:30.965413    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:31.036350 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:28:31.036261    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:28:31.466526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.466492    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"]
Nov 01 12:28:31.473482 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:28:31.473455    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9bd825_433d_4e5f_9bd5_326e83cea7fb.slice/crio-cea553da1f5cb7233be9b401e23c2f3efa369db2b2ac9b77388dd79e1368d348 WatchSource:0}: Error finding container cea553da1f5cb7233be9b401e23c2f3efa369db2b2ac9b77388dd79e1368d348: Status 404 returned error can't find the container with id cea553da1f5cb7233be9b401e23c2f3efa369db2b2ac9b77388dd79e1368d348
Nov 01 12:28:31.713228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.713192    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"]
Nov 01 12:28:31.717713 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.717651    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"
Nov 01 12:28:31.772707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.772644    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zm6rt\""
Nov 01 12:28:31.772874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.772705    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 12:28:31.772874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.772713    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 12:28:31.774446 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.774422    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"]
Nov 01 12:28:31.808685 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.808647    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs" event={"ID":"1a9bd825-433d-4e5f-9bd5-326e83cea7fb","Type":"ContainerStarted","Data":"cea553da1f5cb7233be9b401e23c2f3efa369db2b2ac9b77388dd79e1368d348"}
Nov 01 12:28:31.817104 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.817072    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgrw\" (UniqueName: \"kubernetes.io/projected/0d67bd5f-fba2-46dd-a457-124bf4b71a5f-kube-api-access-fkgrw\") pod \"authorino-operator-7b7bb4bc77-5d2lp\" (UID: \"0d67bd5f-fba2-46dd-a457-124bf4b71a5f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"
Nov 01 12:28:31.918016 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.917972    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgrw\" (UniqueName: \"kubernetes.io/projected/0d67bd5f-fba2-46dd-a457-124bf4b71a5f-kube-api-access-fkgrw\") pod \"authorino-operator-7b7bb4bc77-5d2lp\" (UID: \"0d67bd5f-fba2-46dd-a457-124bf4b71a5f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"
Nov 01 12:28:31.981838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:31.981804    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgrw\" (UniqueName: \"kubernetes.io/projected/0d67bd5f-fba2-46dd-a457-124bf4b71a5f-kube-api-access-fkgrw\") pod \"authorino-operator-7b7bb4bc77-5d2lp\" (UID: \"0d67bd5f-fba2-46dd-a457-124bf4b71a5f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"
Nov 01 12:28:32.030583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:32.030546    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"
Nov 01 12:28:32.411090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:32.410998    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"]
Nov 01 12:28:32.419071 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:28:32.419029    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d67bd5f_fba2_46dd_a457_124bf4b71a5f.slice/crio-eb51aa382ed40bc27f0caec824b40e24f45d211160c02de1dcbb657f4ad0f1b1 WatchSource:0}: Error finding container eb51aa382ed40bc27f0caec824b40e24f45d211160c02de1dcbb657f4ad0f1b1: Status 404 returned error can't find the container with id eb51aa382ed40bc27f0caec824b40e24f45d211160c02de1dcbb657f4ad0f1b1
Nov 01 12:28:32.815958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:32.815923    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp" event={"ID":"0d67bd5f-fba2-46dd-a457-124bf4b71a5f","Type":"ContainerStarted","Data":"eb51aa382ed40bc27f0caec824b40e24f45d211160c02de1dcbb657f4ad0f1b1"}
Nov 01 12:28:35.400978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:35.400935    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"]
Nov 01 12:28:35.405093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:35.405068    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"
Nov 01 12:28:35.450380 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:35.450338    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8gqk\" (UniqueName: \"kubernetes.io/projected/6483676f-1bb7-4013-8246-aa877be51bb5-kube-api-access-k8gqk\") pod \"dns-operator-controller-manager-8557cbdb84-zvk2f\" (UID: \"6483676f-1bb7-4013-8246-aa877be51bb5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"
Nov 01 12:28:35.454734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:35.454702    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 01 12:28:35.501874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:35.501838    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vzjhl\""
Nov 01 12:28:35.504242 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:35.504207    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"]
Nov 01 12:28:35.551239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:35.551204    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8gqk\" (UniqueName: \"kubernetes.io/projected/6483676f-1bb7-4013-8246-aa877be51bb5-kube-api-access-k8gqk\") pod \"dns-operator-controller-manager-8557cbdb84-zvk2f\" (UID: \"6483676f-1bb7-4013-8246-aa877be51bb5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"
Nov 01 12:28:35.607976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:35.607932    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8gqk\" (UniqueName: \"kubernetes.io/projected/6483676f-1bb7-4013-8246-aa877be51bb5-kube-api-access-k8gqk\") pod \"dns-operator-controller-manager-8557cbdb84-zvk2f\" (UID: \"6483676f-1bb7-4013-8246-aa877be51bb5\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"
Nov 01 12:28:35.722901 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:35.722861    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"
Nov 01 12:28:36.126445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:36.126362    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"]
Nov 01 12:28:36.134609 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:28:36.134577    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6483676f_1bb7_4013_8246_aa877be51bb5.slice/crio-a2616cea94eb98ea1da035714b4325c498ad94a6ee352aacf629cab5824c96ba WatchSource:0}: Error finding container a2616cea94eb98ea1da035714b4325c498ad94a6ee352aacf629cab5824c96ba: Status 404 returned error can't find the container with id a2616cea94eb98ea1da035714b4325c498ad94a6ee352aacf629cab5824c96ba
Nov 01 12:28:36.840745 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:36.840654    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp" event={"ID":"0d67bd5f-fba2-46dd-a457-124bf4b71a5f","Type":"ContainerStarted","Data":"c63394115343b4013f19d1e404c0f118a59313ab7a4f7822e0d75a08866f7fba"}
Nov 01 12:28:36.840745 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:36.840712    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"
Nov 01 12:28:36.842246 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:36.842223    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f" event={"ID":"6483676f-1bb7-4013-8246-aa877be51bb5","Type":"ContainerStarted","Data":"a2616cea94eb98ea1da035714b4325c498ad94a6ee352aacf629cab5824c96ba"}
Nov 01 12:28:36.844008 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:36.843981    5725 generic.go:356] "Generic (PLEG): container finished" podID="1a9bd825-433d-4e5f-9bd5-326e83cea7fb" containerID="39286eb39fa600b500e92514b7398abbed79c0e15cb99a303c1429a611c940a7" exitCode=0
Nov 01 12:28:36.844098 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:36.844015    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs" event={"ID":"1a9bd825-433d-4e5f-9bd5-326e83cea7fb","Type":"ContainerDied","Data":"39286eb39fa600b500e92514b7398abbed79c0e15cb99a303c1429a611c940a7"}
Nov 01 12:28:36.952671 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:36.952616    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp" podStartSLOduration=1.70570281 podStartE2EDuration="5.952600803s" totalImagesPullingTime="4.246897993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:36.950903798 +0000 UTC m=+358.458095042" watchObservedRunningTime="2026-11-01 12:28:36.952600803 +0000 UTC m=+358.459792016"
Nov 01 12:28:37.852076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:37.852019    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs" event={"ID":"1a9bd825-433d-4e5f-9bd5-326e83cea7fb","Type":"ContainerStarted","Data":"9955dead91b5f4697c24f6805a8674ed31229d98eb7df3ae704fa498f06974a1"}
Nov 01 12:28:37.852514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:37.852290    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:38.077823 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:38.077770    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs" podStartSLOduration=2.293882198 podStartE2EDuration="8.077754285s" totalImagesPullingTime="5.121544692s" totalInitContainerRuntime="662.327395ms" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:38.030332192 +0000 UTC m=+359.537523419" watchObservedRunningTime="2026-11-01 12:28:38.077754285 +0000 UTC m=+359.584945502"
Nov 01 12:28:39.837131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:39.837101    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 01 12:28:40.330407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.330371    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"]
Nov 01 12:28:40.333602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.333569    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"
Nov 01 12:28:40.399711 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.399677    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t24nx\" (UniqueName: \"kubernetes.io/projected/73be14a6-2a72-424c-9fc9-4f5cc6fd9e53-kube-api-access-t24nx\") pod \"limitador-operator-controller-manager-5778bb4449-4nhzf\" (UID: \"73be14a6-2a72-424c-9fc9-4f5cc6fd9e53\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"
Nov 01 12:28:40.431212 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.431183    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bdwl8\""
Nov 01 12:28:40.432837 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.432817    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"]
Nov 01 12:28:40.500672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.500637    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t24nx\" (UniqueName: \"kubernetes.io/projected/73be14a6-2a72-424c-9fc9-4f5cc6fd9e53-kube-api-access-t24nx\") pod \"limitador-operator-controller-manager-5778bb4449-4nhzf\" (UID: \"73be14a6-2a72-424c-9fc9-4f5cc6fd9e53\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"
Nov 01 12:28:40.554417 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.554389    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t24nx\" (UniqueName: \"kubernetes.io/projected/73be14a6-2a72-424c-9fc9-4f5cc6fd9e53-kube-api-access-t24nx\") pod \"limitador-operator-controller-manager-5778bb4449-4nhzf\" (UID: \"73be14a6-2a72-424c-9fc9-4f5cc6fd9e53\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"
Nov 01 12:28:40.645722 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.645652    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"
Nov 01 12:28:40.869072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.869004    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f" event={"ID":"6483676f-1bb7-4013-8246-aa877be51bb5","Type":"ContainerStarted","Data":"538b8852531b42f48bf3a52a51fd367a2e47b255005fc6c4f3324b5dc17e22a8"}
Nov 01 12:28:40.869545 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.869232    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"
Nov 01 12:28:40.981703 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:40.981650    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f" podStartSLOduration=2.330973818 podStartE2EDuration="5.981635101s" totalImagesPullingTime="3.650661283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:40.980490022 +0000 UTC m=+362.487681239" watchObservedRunningTime="2026-11-01 12:28:40.981635101 +0000 UTC m=+362.488826317"
Nov 01 12:28:41.017933 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:41.017903    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"]
Nov 01 12:28:41.065368 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:28:41.065337    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73be14a6_2a72_424c_9fc9_4f5cc6fd9e53.slice/crio-fdb0ad2d663fb03ec02a0cb2e7115fb6a0561e9aae557577460e731f56aa3b24 WatchSource:0}: Error finding container fdb0ad2d663fb03ec02a0cb2e7115fb6a0561e9aae557577460e731f56aa3b24: Status 404 returned error can't find the container with id fdb0ad2d663fb03ec02a0cb2e7115fb6a0561e9aae557577460e731f56aa3b24
Nov 01 12:28:41.875319 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:41.875277    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf" event={"ID":"73be14a6-2a72-424c-9fc9-4f5cc6fd9e53","Type":"ContainerStarted","Data":"fdb0ad2d663fb03ec02a0cb2e7115fb6a0561e9aae557577460e731f56aa3b24"}
Nov 01 12:28:42.893870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:42.893833    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"]
Nov 01 12:28:42.900475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:42.900456    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"
Nov 01 12:28:42.943678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:42.943643    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"]
Nov 01 12:28:42.987790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:42.987746    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:28:42.988013 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:42.987992    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 01 12:28:42.988092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:42.988004    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8nvb2\""
Nov 01 12:28:43.024076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:43.024025    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a35c7036-2900-4f47-b202-be5c30fef2cd-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-z2shq\" (UID: \"a35c7036-2900-4f47-b202-be5c30fef2cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"
Nov 01 12:28:43.024244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:43.024151    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ss78\" (UniqueName: \"kubernetes.io/projected/a35c7036-2900-4f47-b202-be5c30fef2cd-kube-api-access-7ss78\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-z2shq\" (UID: \"a35c7036-2900-4f47-b202-be5c30fef2cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"
Nov 01 12:28:43.125232 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:43.125194    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a35c7036-2900-4f47-b202-be5c30fef2cd-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-z2shq\" (UID: \"a35c7036-2900-4f47-b202-be5c30fef2cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"
Nov 01 12:28:43.125427 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:43.125290    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ss78\" (UniqueName: \"kubernetes.io/projected/a35c7036-2900-4f47-b202-be5c30fef2cd-kube-api-access-7ss78\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-z2shq\" (UID: \"a35c7036-2900-4f47-b202-be5c30fef2cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"
Nov 01 12:28:43.125598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:43.125574    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a35c7036-2900-4f47-b202-be5c30fef2cd-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-z2shq\" (UID: \"a35c7036-2900-4f47-b202-be5c30fef2cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"
Nov 01 12:28:43.180748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:43.180667    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ss78\" (UniqueName: \"kubernetes.io/projected/a35c7036-2900-4f47-b202-be5c30fef2cd-kube-api-access-7ss78\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-z2shq\" (UID: \"a35c7036-2900-4f47-b202-be5c30fef2cd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"
Nov 01 12:28:43.212583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:43.212541    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"
Nov 01 12:28:43.651438 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:43.651400    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq"]
Nov 01 12:28:43.660264 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:28:43.660227    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35c7036_2900_4f47_b202_be5c30fef2cd.slice/crio-9ac49557d92d3c5b9193b31a522f8d71b7853215ed517ec2f4b0dbda0257bb6a WatchSource:0}: Error finding container 9ac49557d92d3c5b9193b31a522f8d71b7853215ed517ec2f4b0dbda0257bb6a: Status 404 returned error can't find the container with id 9ac49557d92d3c5b9193b31a522f8d71b7853215ed517ec2f4b0dbda0257bb6a
Nov 01 12:28:43.886126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:43.886085    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq" event={"ID":"a35c7036-2900-4f47-b202-be5c30fef2cd","Type":"ContainerStarted","Data":"9ac49557d92d3c5b9193b31a522f8d71b7853215ed517ec2f4b0dbda0257bb6a"}
Nov 01 12:28:44.893597 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:44.893559    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf" event={"ID":"73be14a6-2a72-424c-9fc9-4f5cc6fd9e53","Type":"ContainerStarted","Data":"ba4cde843f43dc3490e34b67de9ef5e38f322f9145bf6cd5d6390ef7d57e750b"}
Nov 01 12:28:44.894004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:44.893650    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"
Nov 01 12:28:45.001959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:45.001900    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf" podStartSLOduration=2.126021641 podStartE2EDuration="5.001887353s" totalImagesPullingTime="2.875865712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:45.000646411 +0000 UTC m=+366.507837650" watchObservedRunningTime="2026-11-01 12:28:45.001887353 +0000 UTC m=+366.509078567"
Nov 01 12:28:46.905982 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:46.905948    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq" event={"ID":"a35c7036-2900-4f47-b202-be5c30fef2cd","Type":"ContainerStarted","Data":"a44c8c26ef2366308e1cd1f236799940063f99c7f10751e518d837d036ff3312"}
Nov 01 12:28:47.125596 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:47.125542    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-z2shq" podStartSLOduration=2.463310574 podStartE2EDuration="5.125526951s" totalImagesPullingTime="2.662216377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:47.077728227 +0000 UTC m=+368.584919457" watchObservedRunningTime="2026-11-01 12:28:47.125526951 +0000 UTC m=+368.632718169"
Nov 01 12:28:47.854484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:47.854454    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5d2lp"
Nov 01 12:28:48.431531 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.431489    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"]
Nov 01 12:28:48.435248 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.435223    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:28:48.488501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.488476    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xwdxs\""
Nov 01 12:28:48.537628 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.537599    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"]
Nov 01 12:28:48.571170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.571145    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqq8f\" (UniqueName: \"kubernetes.io/projected/96289b27-9274-4f9f-9023-3173737f12e3-kube-api-access-pqq8f\") pod \"kuadrant-operator-controller-manager-678d87766f-tj76t\" (UID: \"96289b27-9274-4f9f-9023-3173737f12e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:28:48.571317 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.571181    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96289b27-9274-4f9f-9023-3173737f12e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tj76t\" (UID: \"96289b27-9274-4f9f-9023-3173737f12e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:28:48.671949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.671918    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqq8f\" (UniqueName: \"kubernetes.io/projected/96289b27-9274-4f9f-9023-3173737f12e3-kube-api-access-pqq8f\") pod \"kuadrant-operator-controller-manager-678d87766f-tj76t\" (UID: \"96289b27-9274-4f9f-9023-3173737f12e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:28:48.672147 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.671968    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96289b27-9274-4f9f-9023-3173737f12e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tj76t\" (UID: \"96289b27-9274-4f9f-9023-3173737f12e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:28:48.672432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.672413    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96289b27-9274-4f9f-9023-3173737f12e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tj76t\" (UID: \"96289b27-9274-4f9f-9023-3173737f12e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:28:48.729198 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.729164    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqq8f\" (UniqueName: \"kubernetes.io/projected/96289b27-9274-4f9f-9023-3173737f12e3-kube-api-access-pqq8f\") pod \"kuadrant-operator-controller-manager-678d87766f-tj76t\" (UID: \"96289b27-9274-4f9f-9023-3173737f12e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:28:48.747530 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.747504    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:28:48.859529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:48.859502    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-9cdcs"
Nov 01 12:28:49.179882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:49.179804    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"]
Nov 01 12:28:49.188484 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:28:49.188454    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96289b27_9274_4f9f_9023_3173737f12e3.slice/crio-0b140c6e37c52e57ecfcd9dd2b063d4abe03e2c7c013300d83603b076a6359cf WatchSource:0}: Error finding container 0b140c6e37c52e57ecfcd9dd2b063d4abe03e2c7c013300d83603b076a6359cf: Status 404 returned error can't find the container with id 0b140c6e37c52e57ecfcd9dd2b063d4abe03e2c7c013300d83603b076a6359cf
Nov 01 12:28:49.920311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:49.920277    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t" event={"ID":"96289b27-9274-4f9f-9023-3173737f12e3","Type":"ContainerStarted","Data":"0b140c6e37c52e57ecfcd9dd2b063d4abe03e2c7c013300d83603b076a6359cf"}
Nov 01 12:28:51.878318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:51.878280    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zvk2f"
Nov 01 12:28:54.949788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:54.949746    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t" event={"ID":"96289b27-9274-4f9f-9023-3173737f12e3","Type":"ContainerStarted","Data":"16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0"}
Nov 01 12:28:54.950228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:54.949878    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:28:55.153790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:55.153740    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t" podStartSLOduration=2.040121758 podStartE2EDuration="7.153726538s" totalImagesPullingTime="5.11360478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:55.153574328 +0000 UTC m=+376.660765545" watchObservedRunningTime="2026-11-01 12:28:55.153726538 +0000 UTC m=+376.660917755"
Nov 01 12:28:55.902044 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:28:55.902010    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4nhzf"
Nov 01 12:29:01.027217 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:29:01.027176    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:29:05.439313 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.439277    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"]
Nov 01 12:29:05.444130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.444107    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.516184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.516140    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"]
Nov 01 12:29:05.526686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.526652    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmnnq\" (UniqueName: \"kubernetes.io/projected/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-kube-api-access-hmnnq\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.526824 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.526700    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.526824 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.526806    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.526896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.526830    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.526896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.526860    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.526959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.526908    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.526959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.526934    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.527028 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.527006    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.527089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.527061    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.542374 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.542345    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-j5vc8\""
Nov 01 12:29:05.628003 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.627966    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628014    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628039    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628175    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628228    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmnnq\" (UniqueName: \"kubernetes.io/projected/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-kube-api-access-hmnnq\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628269    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628312    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628347    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628388    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628880    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.628905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628900    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.629082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628972    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.629082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.628998    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.629329 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.629307    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.630621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.630599    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.630785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.630764    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.682456 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.682417    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.682662 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.682645    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmnnq\" (UniqueName: \"kubernetes.io/projected/7774276a-5194-4bbc-bb2b-7eb93e7fa6c2-kube-api-access-hmnnq\") pod \"data-science-gateway-data-science-gateway-class-575956cc9clm8lm\" (UID: \"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.757773 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.757735    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:05.957442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:05.957410    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 12:29:06.190899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:06.190811    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"]
Nov 01 12:29:06.199831 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:29:06.199801    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7774276a_5194_4bbc_bb2b_7eb93e7fa6c2.slice/crio-4c97039ae8db6246ed2fa0b55090aee5f7ccc47bbdfa051b5bf82422f404266f WatchSource:0}: Error finding container 4c97039ae8db6246ed2fa0b55090aee5f7ccc47bbdfa051b5bf82422f404266f: Status 404 returned error can't find the container with id 4c97039ae8db6246ed2fa0b55090aee5f7ccc47bbdfa051b5bf82422f404266f
Nov 01 12:29:06.202478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:06.202439    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:29:06.202559 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:06.202525    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:29:06.202605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:06.202591    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:29:07.022073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:07.022010    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm" event={"ID":"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2","Type":"ContainerStarted","Data":"bf64abbf42f12c3087d7183dd7284ee94ab184b2f967a5cec8dfd1e3bc75c1cd"}
Nov 01 12:29:07.022565 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:07.022114    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm" event={"ID":"7774276a-5194-4bbc-bb2b-7eb93e7fa6c2","Type":"ContainerStarted","Data":"4c97039ae8db6246ed2fa0b55090aee5f7ccc47bbdfa051b5bf82422f404266f"}
Nov 01 12:29:07.142683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:07.142629    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm" podStartSLOduration=2.142615608 podStartE2EDuration="2.142615608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:07.13996016 +0000 UTC m=+388.647151376" watchObservedRunningTime="2026-11-01 12:29:07.142615608 +0000 UTC m=+388.649806825"
Nov 01 12:29:07.758411 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:07.758381    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:07.762987 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:07.762965    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:08.027605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:08.027521    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:08.028589 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:08.028566    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9clm8lm"
Nov 01 12:29:08.888165 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:08.888130    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h968c"]
Nov 01 12:29:08.894868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:08.894837    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h968c"
Nov 01 12:29:08.944293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:08.944265    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kj2v2\""
Nov 01 12:29:08.945489 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:08.945459    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h968c"]
Nov 01 12:29:09.057653 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:09.057622    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdh6x\" (UniqueName: \"kubernetes.io/projected/f2c91aae-4860-4943-999b-4a409faf3342-kube-api-access-bdh6x\") pod \"authorino-57f7f8659f-h968c\" (UID: \"f2c91aae-4860-4943-999b-4a409faf3342\") " pod="kuadrant-system/authorino-57f7f8659f-h968c"
Nov 01 12:29:09.159144 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:09.159040    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdh6x\" (UniqueName: \"kubernetes.io/projected/f2c91aae-4860-4943-999b-4a409faf3342-kube-api-access-bdh6x\") pod \"authorino-57f7f8659f-h968c\" (UID: \"f2c91aae-4860-4943-999b-4a409faf3342\") " pod="kuadrant-system/authorino-57f7f8659f-h968c"
Nov 01 12:29:09.214729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:09.214692    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdh6x\" (UniqueName: \"kubernetes.io/projected/f2c91aae-4860-4943-999b-4a409faf3342-kube-api-access-bdh6x\") pod \"authorino-57f7f8659f-h968c\" (UID: \"f2c91aae-4860-4943-999b-4a409faf3342\") " pod="kuadrant-system/authorino-57f7f8659f-h968c"
Nov 01 12:29:09.217316 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:09.217293    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h968c"
Nov 01 12:29:09.592146 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:09.592108    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h968c"]
Nov 01 12:29:09.599729 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:29:09.599698    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2c91aae_4860_4943_999b_4a409faf3342.slice/crio-80dc96c75737a01580696052ffd2660d84846885a2f23467f7f58ec06ab1593b WatchSource:0}: Error finding container 80dc96c75737a01580696052ffd2660d84846885a2f23467f7f58ec06ab1593b: Status 404 returned error can't find the container with id 80dc96c75737a01580696052ffd2660d84846885a2f23467f7f58ec06ab1593b
Nov 01 12:29:10.039523 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:10.039486    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h968c" event={"ID":"f2c91aae-4860-4943-999b-4a409faf3342","Type":"ContainerStarted","Data":"80dc96c75737a01580696052ffd2660d84846885a2f23467f7f58ec06ab1593b"}
Nov 01 12:29:11.556446 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:11.556412    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2tmwx"]
Nov 01 12:29:11.560240 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:11.560216    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx"
Nov 01 12:29:11.625085 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:11.625032    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2tmwx"]
Nov 01 12:29:11.681996 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:11.681953    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4c5z\" (UniqueName: \"kubernetes.io/projected/3bd9b065-d579-48ad-890a-61a43423e20f-kube-api-access-l4c5z\") pod \"authorino-56c6c9ddd5-2tmwx\" (UID: \"3bd9b065-d579-48ad-890a-61a43423e20f\") " pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx"
Nov 01 12:29:11.782520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:11.782483    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4c5z\" (UniqueName: \"kubernetes.io/projected/3bd9b065-d579-48ad-890a-61a43423e20f-kube-api-access-l4c5z\") pod \"authorino-56c6c9ddd5-2tmwx\" (UID: \"3bd9b065-d579-48ad-890a-61a43423e20f\") " pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx"
Nov 01 12:29:11.836433 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:11.836368    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4c5z\" (UniqueName: \"kubernetes.io/projected/3bd9b065-d579-48ad-890a-61a43423e20f-kube-api-access-l4c5z\") pod \"authorino-56c6c9ddd5-2tmwx\" (UID: \"3bd9b065-d579-48ad-890a-61a43423e20f\") " pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx"
Nov 01 12:29:11.874373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:11.874108    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx"
Nov 01 12:29:12.253722 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:12.253684    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2tmwx"]
Nov 01 12:29:12.261832 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:29:12.261787    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bd9b065_d579_48ad_890a_61a43423e20f.slice/crio-6f1801fa4896739f7aea1eb6ef9a0e36bbaf70c1714c516bab9f67f17d7a72ad WatchSource:0}: Error finding container 6f1801fa4896739f7aea1eb6ef9a0e36bbaf70c1714c516bab9f67f17d7a72ad: Status 404 returned error can't find the container with id 6f1801fa4896739f7aea1eb6ef9a0e36bbaf70c1714c516bab9f67f17d7a72ad
Nov 01 12:29:13.058042 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:13.058001    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx" event={"ID":"3bd9b065-d579-48ad-890a-61a43423e20f","Type":"ContainerStarted","Data":"6f1801fa4896739f7aea1eb6ef9a0e36bbaf70c1714c516bab9f67f17d7a72ad"}
Nov 01 12:29:13.059714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:13.059636    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h968c" event={"ID":"f2c91aae-4860-4943-999b-4a409faf3342","Type":"ContainerStarted","Data":"fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92"}
Nov 01 12:29:13.178563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:13.178502    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-h968c" podStartSLOduration=2.221629235 podStartE2EDuration="5.17848485s" totalImagesPullingTime="2.956855615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:13.177653615 +0000 UTC m=+394.684844832" watchObservedRunningTime="2026-11-01 12:29:13.17848485 +0000 UTC m=+394.685676068"
Nov 01 12:29:14.067546 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:14.067494    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx" event={"ID":"3bd9b065-d579-48ad-890a-61a43423e20f","Type":"ContainerStarted","Data":"062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1"}
Nov 01 12:29:14.173994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:14.173890    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx" podStartSLOduration=1.56992025 podStartE2EDuration="3.173873644s" totalImagesPullingTime="1.603953394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:14.172642929 +0000 UTC m=+395.679834143" watchObservedRunningTime="2026-11-01 12:29:14.173873644 +0000 UTC m=+395.681064862"
Nov 01 12:29:14.195276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:14.195244    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h968c"]
Nov 01 12:29:15.075956 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:15.075910    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-h968c" podUID="f2c91aae-4860-4943-999b-4a409faf3342" containerName="authorino" containerID="cri-o://fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92" gracePeriod=30
Nov 01 12:29:15.316530 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:15.316508    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h968c"
Nov 01 12:29:15.416900 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:15.416809    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c91aae-4860-4943-999b-4a409faf3342-kube-api-access-bdh6x\" (UniqueName: \"kubernetes.io/projected/f2c91aae-4860-4943-999b-4a409faf3342-kube-api-access-bdh6x\") pod \"f2c91aae-4860-4943-999b-4a409faf3342\" (UID: \"f2c91aae-4860-4943-999b-4a409faf3342\") "
Nov 01 12:29:15.418944 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:15.418915    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c91aae-4860-4943-999b-4a409faf3342-kube-api-access-bdh6x" pod "f2c91aae-4860-4943-999b-4a409faf3342" (UID: "f2c91aae-4860-4943-999b-4a409faf3342"). InnerVolumeSpecName "kube-api-access-bdh6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:29:15.517763 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:15.517729    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdh6x\" (UniqueName: \"kubernetes.io/projected/f2c91aae-4860-4943-999b-4a409faf3342-kube-api-access-bdh6x\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:29:16.082481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.082443    5725 generic.go:356] "Generic (PLEG): container finished" podID="f2c91aae-4860-4943-999b-4a409faf3342" containerID="fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92" exitCode=0
Nov 01 12:29:16.082970 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.082492    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h968c"
Nov 01 12:29:16.082970 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.082525    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h968c" event={"ID":"f2c91aae-4860-4943-999b-4a409faf3342","Type":"ContainerDied","Data":"fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92"}
Nov 01 12:29:16.082970 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.082569    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h968c" event={"ID":"f2c91aae-4860-4943-999b-4a409faf3342","Type":"ContainerDied","Data":"80dc96c75737a01580696052ffd2660d84846885a2f23467f7f58ec06ab1593b"}
Nov 01 12:29:16.082970 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.082588    5725 scope.go:122] "RemoveContainer" containerID="fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92"
Nov 01 12:29:16.091514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.091485    5725 scope.go:122] "RemoveContainer" containerID="fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92"
Nov 01 12:29:16.091808 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:29:16.091786    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92\": container with ID starting with fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92 not found: ID does not exist" containerID="fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92"
Nov 01 12:29:16.091887 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.091819    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92"} err="failed to get container status \"fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92\": rpc error: code = NotFound desc = could not find container \"fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92\": container with ID starting with fd4488f940283221bbd1c453a2af39f288b945fdf8ac0c223302144e74b59b92 not found: ID does not exist"
Nov 01 12:29:16.235373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.235340    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h968c"]
Nov 01 12:29:16.238598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.238570    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h968c"]
Nov 01 12:29:16.614795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:16.614763    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c91aae-4860-4943-999b-4a409faf3342" path="/var/lib/kubelet/pods/f2c91aae-4860-4943-999b-4a409faf3342/volumes"
Nov 01 12:29:17.997968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:17.997934    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"]
Nov 01 12:29:17.998496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:17.998476    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f2c91aae-4860-4943-999b-4a409faf3342" containerName="authorino"
Nov 01 12:29:17.998540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:17.998499    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c91aae-4860-4943-999b-4a409faf3342" containerName="authorino"
Nov 01 12:29:17.998604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:17.998594    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c91aae-4860-4943-999b-4a409faf3342" containerName="authorino"
Nov 01 12:29:18.003334 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.003317    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.037189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.037155    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac494857-512e-4d8d-ae3e-127a0ededaf2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.037328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.037263    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac494857-512e-4d8d-ae3e-127a0ededaf2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.037328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.037295    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ac494857-512e-4d8d-ae3e-127a0ededaf2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.037328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.037314    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxk62\" (UniqueName: \"kubernetes.io/projected/ac494857-512e-4d8d-ae3e-127a0ededaf2-kube-api-access-pxk62\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.052682 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.052653    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 12:29:18.099462 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.099431    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 12:29:18.099623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.099522    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-jlh2s\""
Nov 01 12:29:18.137935 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.137899    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac494857-512e-4d8d-ae3e-127a0ededaf2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.138089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.137993    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac494857-512e-4d8d-ae3e-127a0ededaf2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.138089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.138023    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ac494857-512e-4d8d-ae3e-127a0ededaf2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.138089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.138063    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxk62\" (UniqueName: \"kubernetes.io/projected/ac494857-512e-4d8d-ae3e-127a0ededaf2-kube-api-access-pxk62\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.138427 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.138408    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac494857-512e-4d8d-ae3e-127a0ededaf2-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.138462 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.138426    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ac494857-512e-4d8d-ae3e-127a0ededaf2-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.194238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.194184    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 12:29:18.196354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.196322    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"]
Nov 01 12:29:18.196635 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.196615    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxk62\" (UniqueName: \"kubernetes.io/projected/ac494857-512e-4d8d-ae3e-127a0ededaf2-kube-api-access-pxk62\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.198887 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.198868    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac494857-512e-4d8d-ae3e-127a0ededaf2-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tlnw9\" (UID: \"ac494857-512e-4d8d-ae3e-127a0ededaf2\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.322344 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.322257    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:18.746834 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:18.746802    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"]
Nov 01 12:29:18.754688 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:29:18.754660    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac494857_512e_4d8d_ae3e_127a0ededaf2.slice/crio-8d4f16a238134c919bfdd317da1b9b3460f3e4b6f118dfadc2cfbdcc033e0e8a WatchSource:0}: Error finding container 8d4f16a238134c919bfdd317da1b9b3460f3e4b6f118dfadc2cfbdcc033e0e8a: Status 404 returned error can't find the container with id 8d4f16a238134c919bfdd317da1b9b3460f3e4b6f118dfadc2cfbdcc033e0e8a
Nov 01 12:29:19.101668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:19.101586    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9" event={"ID":"ac494857-512e-4d8d-ae3e-127a0ededaf2","Type":"ContainerStarted","Data":"8d4f16a238134c919bfdd317da1b9b3460f3e4b6f118dfadc2cfbdcc033e0e8a"}
Nov 01 12:29:22.121027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:22.120940    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9" event={"ID":"ac494857-512e-4d8d-ae3e-127a0ededaf2","Type":"ContainerStarted","Data":"3738d71feed6ec7da46a093d2c019240363ade96b2d0997a7df87ecbe6405c8f"}
Nov 01 12:29:22.121440 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:22.121038    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:22.278993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:22.278939    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9" podStartSLOduration=2.276233906 podStartE2EDuration="5.27892589s" totalImagesPullingTime="3.002691984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:22.278520625 +0000 UTC m=+403.785711834" watchObservedRunningTime="2026-11-01 12:29:22.27892589 +0000 UTC m=+403.786117106"
Nov 01 12:29:30.948973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:30.948934    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"]
Nov 01 12:29:30.952518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:30.952497    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:31.004836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.004808    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"]
Nov 01 12:29:31.026460 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:29:31.026420    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:29:31.049503 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.049478    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 12:29:31.049503 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.049490    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 12:29:31.051867 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.051847    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/485e0b05-d222-4063-82fe-5211239527db-data\") pod \"postgres-7cdbf6dc44-ctvqj\" (UID: \"485e0b05-d222-4063-82fe-5211239527db\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:31.051966 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.051952    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp96f\" (UniqueName: \"kubernetes.io/projected/485e0b05-d222-4063-82fe-5211239527db-kube-api-access-wp96f\") pod \"postgres-7cdbf6dc44-ctvqj\" (UID: \"485e0b05-d222-4063-82fe-5211239527db\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:31.096769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.096734    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wbrhh\""
Nov 01 12:29:31.096769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.096763    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 01 12:29:31.152396 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.152354    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/485e0b05-d222-4063-82fe-5211239527db-data\") pod \"postgres-7cdbf6dc44-ctvqj\" (UID: \"485e0b05-d222-4063-82fe-5211239527db\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:31.152571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.152451    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp96f\" (UniqueName: \"kubernetes.io/projected/485e0b05-d222-4063-82fe-5211239527db-kube-api-access-wp96f\") pod \"postgres-7cdbf6dc44-ctvqj\" (UID: \"485e0b05-d222-4063-82fe-5211239527db\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:31.152766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.152749    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/485e0b05-d222-4063-82fe-5211239527db-data\") pod \"postgres-7cdbf6dc44-ctvqj\" (UID: \"485e0b05-d222-4063-82fe-5211239527db\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:31.206574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.206504    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp96f\" (UniqueName: \"kubernetes.io/projected/485e0b05-d222-4063-82fe-5211239527db-kube-api-access-wp96f\") pod \"postgres-7cdbf6dc44-ctvqj\" (UID: \"485e0b05-d222-4063-82fe-5211239527db\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:31.265117 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.265081    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:31.637579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:31.637547    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"]
Nov 01 12:29:31.645265 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:29:31.645236    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485e0b05_d222_4063_82fe_5211239527db.slice/crio-91e3c5e7c3159cfe35f24005f2c1a34de87b7c948eb96258e7a2e213721ce750 WatchSource:0}: Error finding container 91e3c5e7c3159cfe35f24005f2c1a34de87b7c948eb96258e7a2e213721ce750: Status 404 returned error can't find the container with id 91e3c5e7c3159cfe35f24005f2c1a34de87b7c948eb96258e7a2e213721ce750
Nov 01 12:29:32.177528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:32.177490    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj" event={"ID":"485e0b05-d222-4063-82fe-5211239527db","Type":"ContainerStarted","Data":"91e3c5e7c3159cfe35f24005f2c1a34de87b7c948eb96258e7a2e213721ce750"}
Nov 01 12:29:39.219936 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:39.219888    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj" event={"ID":"485e0b05-d222-4063-82fe-5211239527db","Type":"ContainerStarted","Data":"711abd8849bf84d69f9a57bc57854834d75d40c20540cfc51ee307e05a3c6d34"}
Nov 01 12:29:39.220354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:39.219980    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:39.341251 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:39.341197    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj" podStartSLOduration=2.542606005 podStartE2EDuration="9.341184732s" totalImagesPullingTime="6.798578727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:39.33839721 +0000 UTC m=+420.845588423" watchObservedRunningTime="2026-11-01 12:29:39.341184732 +0000 UTC m=+420.848375949"
Nov 01 12:29:43.127543 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:43.127512    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tlnw9"
Nov 01 12:29:45.255491 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:45.255459    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ctvqj"
Nov 01 12:29:51.968113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:51.968075    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s4fkl"]
Nov 01 12:29:51.971668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:51.971646    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.044394 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.044356    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-webhook-cert\") pod \"maas-controller-579cf7c6d-s4fkl\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") " pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.044573 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.044419    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqq2\" (UniqueName: \"kubernetes.io/projected/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-kube-api-access-9gqq2\") pod \"maas-controller-579cf7c6d-s4fkl\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") " pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.044573 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.044560    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-metrics-certs\") pod \"maas-controller-579cf7c6d-s4fkl\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") " pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.067831 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.067798    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 12:29:52.067831 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.067822    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 12:29:52.068020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.067811    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-q9pth\""
Nov 01 12:29:52.068020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.067893    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 12:29:52.070224 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.070192    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s4fkl"]
Nov 01 12:29:52.144995 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.144961    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-metrics-certs\") pod \"maas-controller-579cf7c6d-s4fkl\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") " pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.145207 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.145007    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-webhook-cert\") pod \"maas-controller-579cf7c6d-s4fkl\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") " pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.145207 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.145039    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqq2\" (UniqueName: \"kubernetes.io/projected/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-kube-api-access-9gqq2\") pod \"maas-controller-579cf7c6d-s4fkl\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") " pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.147496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.147469    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-metrics-certs\") pod \"maas-controller-579cf7c6d-s4fkl\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") " pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.147496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.147481    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-webhook-cert\") pod \"maas-controller-579cf7c6d-s4fkl\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") " pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.199773 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.199730    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqq2\" (UniqueName: \"kubernetes.io/projected/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-kube-api-access-9gqq2\") pod \"maas-controller-579cf7c6d-s4fkl\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") " pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.284520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.284488    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:52.356140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.356112    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57458d666c-mbkth"]
Nov 01 12:29:52.360498 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.360474    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.414677 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.414644    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57458d666c-mbkth"]
Nov 01 12:29:52.448365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.448328    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfxm\" (UniqueName: \"kubernetes.io/projected/a52ae595-903b-4c45-9802-e106d8519f07-kube-api-access-bxfxm\") pod \"maas-controller-57458d666c-mbkth\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") " pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.448529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.448398    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-webhook-cert\") pod \"maas-controller-57458d666c-mbkth\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") " pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.448529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.448505    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-metrics-certs\") pod \"maas-controller-57458d666c-mbkth\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") " pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.549479 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.549404    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-webhook-cert\") pod \"maas-controller-57458d666c-mbkth\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") " pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.549479 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.549455    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-metrics-certs\") pod \"maas-controller-57458d666c-mbkth\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") " pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.549716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.549492    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfxm\" (UniqueName: \"kubernetes.io/projected/a52ae595-903b-4c45-9802-e106d8519f07-kube-api-access-bxfxm\") pod \"maas-controller-57458d666c-mbkth\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") " pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.551857 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.551823    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-metrics-certs\") pod \"maas-controller-57458d666c-mbkth\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") " pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.551970 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.551860    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-webhook-cert\") pod \"maas-controller-57458d666c-mbkth\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") " pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.602683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.602649    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfxm\" (UniqueName: \"kubernetes.io/projected/a52ae595-903b-4c45-9802-e106d8519f07-kube-api-access-bxfxm\") pod \"maas-controller-57458d666c-mbkth\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") " pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:52.656266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.656232    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s4fkl"]
Nov 01 12:29:52.663763 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:29:52.663701    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45adcbc_88f1_4a66_8b46_fcbf38ff6d55.slice/crio-7601cdc7dbf657e95765cd438fc18b568a2132ac917b1bfc6b898c384ded2d89 WatchSource:0}: Error finding container 7601cdc7dbf657e95765cd438fc18b568a2132ac917b1bfc6b898c384ded2d89: Status 404 returned error can't find the container with id 7601cdc7dbf657e95765cd438fc18b568a2132ac917b1bfc6b898c384ded2d89
Nov 01 12:29:52.674694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:52.674671    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:53.042915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:53.042882    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57458d666c-mbkth"]
Nov 01 12:29:53.050551 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:29:53.050524    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52ae595_903b_4c45_9802_e106d8519f07.slice/crio-9b5c33da725a9439d39c8bba8827a0b895e3795d70a6913056900a6ecd010288 WatchSource:0}: Error finding container 9b5c33da725a9439d39c8bba8827a0b895e3795d70a6913056900a6ecd010288: Status 404 returned error can't find the container with id 9b5c33da725a9439d39c8bba8827a0b895e3795d70a6913056900a6ecd010288
Nov 01 12:29:53.309612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:53.309520    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mbkth" event={"ID":"a52ae595-903b-4c45-9802-e106d8519f07","Type":"ContainerStarted","Data":"9b5c33da725a9439d39c8bba8827a0b895e3795d70a6913056900a6ecd010288"}
Nov 01 12:29:53.310774 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:53.310738    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-s4fkl" event={"ID":"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55","Type":"ContainerStarted","Data":"7601cdc7dbf657e95765cd438fc18b568a2132ac917b1bfc6b898c384ded2d89"}
Nov 01 12:29:56.337282 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:56.337236    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-s4fkl" event={"ID":"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55","Type":"ContainerStarted","Data":"02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca"}
Nov 01 12:29:56.337640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:56.337520    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:29:56.448713 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:56.448615    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-s4fkl" podStartSLOduration=1.94383331 podStartE2EDuration="5.448598901s" totalImagesPullingTime="3.504765591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:56.44608609 +0000 UTC m=+437.953277307" watchObservedRunningTime="2026-11-01 12:29:56.448598901 +0000 UTC m=+437.955790117"
Nov 01 12:29:57.345331 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:57.345288    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mbkth" event={"ID":"a52ae595-903b-4c45-9802-e106d8519f07","Type":"ContainerStarted","Data":"72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38"}
Nov 01 12:29:57.345861 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:57.345576    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:29:57.453919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:57.453859    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-57458d666c-mbkth" podStartSLOduration=1.9286654030000001 podStartE2EDuration="5.453842229s" totalImagesPullingTime="3.525176826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:57.452729963 +0000 UTC m=+438.959921195" watchObservedRunningTime="2026-11-01 12:29:57.453842229 +0000 UTC m=+438.961033445"
Nov 01 12:29:59.409306 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.409263    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"]
Nov 01 12:29:59.413114 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.413091    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.465013 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.464979    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"]
Nov 01 12:29:59.509737 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.509699    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 01 12:29:59.509737 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.509711    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 01 12:29:59.509949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.509736    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 01 12:29:59.509949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.509765    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qgjg4\""
Nov 01 12:29:59.519239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.519215    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2l5b\" (UniqueName: \"kubernetes.io/projected/6a039198-640d-40f0-9507-2968a5413a77-kube-api-access-w2l5b\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.519363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.519250    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a039198-640d-40f0-9507-2968a5413a77-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.519363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.519270    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a039198-640d-40f0-9507-2968a5413a77-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.519363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.519289    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a039198-640d-40f0-9507-2968a5413a77-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.620216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.620183    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2l5b\" (UniqueName: \"kubernetes.io/projected/6a039198-640d-40f0-9507-2968a5413a77-kube-api-access-w2l5b\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.620375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.620224    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a039198-640d-40f0-9507-2968a5413a77-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.620375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.620258    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a039198-640d-40f0-9507-2968a5413a77-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.620375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.620285    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a039198-640d-40f0-9507-2968a5413a77-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.620481 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:29:59.620373    5725 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 01 12:29:59.620481 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:29:59.620464    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a039198-640d-40f0-9507-2968a5413a77-proxy-source podName:6a039198-640d-40f0-9507-2968a5413a77 nodeName:}" failed. No retries permitted until 2026-11-01 12:30:00.120436016 +0000 UTC m=+441.627627228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/6a039198-640d-40f0-9507-2968a5413a77-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-vfwh2" (UID: "6a039198-640d-40f0-9507-2968a5413a77") : configmap "usage-tenancy-proxy-source" not found
Nov 01 12:29:59.620481 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:29:59.620467    5725 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 01 12:29:59.620573 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:29:59.620533    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a039198-640d-40f0-9507-2968a5413a77-tls-cert podName:6a039198-640d-40f0-9507-2968a5413a77 nodeName:}" failed. No retries permitted until 2026-11-01 12:30:00.120516463 +0000 UTC m=+441.627707678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/6a039198-640d-40f0-9507-2968a5413a77-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-vfwh2" (UID: "6a039198-640d-40f0-9507-2968a5413a77") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 01 12:29:59.620869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.620851    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a039198-640d-40f0-9507-2968a5413a77-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:29:59.674642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:29:59.674574    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2l5b\" (UniqueName: \"kubernetes.io/projected/6a039198-640d-40f0-9507-2968a5413a77-kube-api-access-w2l5b\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:30:00.125433 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:00.125395    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a039198-640d-40f0-9507-2968a5413a77-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:30:00.125640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:00.125495    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a039198-640d-40f0-9507-2968a5413a77-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:30:00.126340 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:00.126319    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a039198-640d-40f0-9507-2968a5413a77-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:30:00.127978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:00.127953    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a039198-640d-40f0-9507-2968a5413a77-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vfwh2\" (UID: \"6a039198-640d-40f0-9507-2968a5413a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:30:00.325360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:00.325331    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:30:00.698921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:00.698844    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"]
Nov 01 12:30:00.705928 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:30:00.705897    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a039198_640d_40f0_9507_2968a5413a77.slice/crio-229e31dc9a2a6eb223e5e1aca28830a6cf24dd174b9ed0dc85c911fe8f590975 WatchSource:0}: Error finding container 229e31dc9a2a6eb223e5e1aca28830a6cf24dd174b9ed0dc85c911fe8f590975: Status 404 returned error can't find the container with id 229e31dc9a2a6eb223e5e1aca28830a6cf24dd174b9ed0dc85c911fe8f590975
Nov 01 12:30:01.025850 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:30:01.025812    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:30:01.368881 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:01.368801    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2" event={"ID":"6a039198-640d-40f0-9507-2968a5413a77","Type":"ContainerStarted","Data":"229e31dc9a2a6eb223e5e1aca28830a6cf24dd174b9ed0dc85c911fe8f590975"}
Nov 01 12:30:07.348921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:07.348199    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:30:08.353806 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:08.353703    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:30:08.512727 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:08.512688    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s4fkl"]
Nov 01 12:30:08.513523 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:08.512914    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-s4fkl" podUID="d45adcbc-88f1-4a66-8b46-fcbf38ff6d55" containerName="manager" containerID="cri-o://02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca" gracePeriod=30
Nov 01 12:30:09.295456 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.295173    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:30:09.322100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.322042    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-metrics-certs\") pod \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") "
Nov 01 12:30:09.322289 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.322137    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-webhook-cert\") pod \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") "
Nov 01 12:30:09.322289 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.322182    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-kube-api-access-9gqq2\" (UniqueName: \"kubernetes.io/projected/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-kube-api-access-9gqq2\") pod \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\" (UID: \"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55\") "
Nov 01 12:30:09.325557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.325514    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-kube-api-access-9gqq2" pod "d45adcbc-88f1-4a66-8b46-fcbf38ff6d55" (UID: "d45adcbc-88f1-4a66-8b46-fcbf38ff6d55"). InnerVolumeSpecName "kube-api-access-9gqq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:30:09.325770 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.325749    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-metrics-certs" pod "d45adcbc-88f1-4a66-8b46-fcbf38ff6d55" (UID: "d45adcbc-88f1-4a66-8b46-fcbf38ff6d55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:30:09.326675 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.326632    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-webhook-cert" pod "d45adcbc-88f1-4a66-8b46-fcbf38ff6d55" (UID: "d45adcbc-88f1-4a66-8b46-fcbf38ff6d55"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:30:09.423088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.422983    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:30:09.423088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.423017    5725 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-webhook-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:30:09.423088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.423031    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gqq2\" (UniqueName: \"kubernetes.io/projected/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55-kube-api-access-9gqq2\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:30:09.423835 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.423800    5725 generic.go:356] "Generic (PLEG): container finished" podID="d45adcbc-88f1-4a66-8b46-fcbf38ff6d55" containerID="02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca" exitCode=0
Nov 01 12:30:09.423960 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.423859    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-s4fkl" event={"ID":"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55","Type":"ContainerDied","Data":"02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca"}
Nov 01 12:30:09.423960 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.423866    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-s4fkl"
Nov 01 12:30:09.423960 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.423916    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-s4fkl" event={"ID":"d45adcbc-88f1-4a66-8b46-fcbf38ff6d55","Type":"ContainerDied","Data":"7601cdc7dbf657e95765cd438fc18b568a2132ac917b1bfc6b898c384ded2d89"}
Nov 01 12:30:09.423960 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.423949    5725 scope.go:122] "RemoveContainer" containerID="02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca"
Nov 01 12:30:09.438993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.438969    5725 scope.go:122] "RemoveContainer" containerID="02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca"
Nov 01 12:30:09.439438 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:30:09.439400    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca\": container with ID starting with 02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca not found: ID does not exist" containerID="02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca"
Nov 01 12:30:09.439527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.439440    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca"} err="failed to get container status \"02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca\": rpc error: code = NotFound desc = could not find container \"02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca\": container with ID starting with 02c89c35be968ccd535b70c85928ed4234cf1e79d3c6e591088759a1bb373dca not found: ID does not exist"
Nov 01 12:30:09.630231 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.630127    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s4fkl"]
Nov 01 12:30:09.632472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:09.632438    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s4fkl"]
Nov 01 12:30:10.617814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:10.617782    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45adcbc-88f1-4a66-8b46-fcbf38ff6d55" path="/var/lib/kubelet/pods/d45adcbc-88f1-4a66-8b46-fcbf38ff6d55/volumes"
Nov 01 12:30:15.469269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:15.469234    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2" event={"ID":"6a039198-640d-40f0-9507-2968a5413a77","Type":"ContainerStarted","Data":"d521994069fcdc905874917c986d4ccc6fd32fc7e2404590beaf036bbf49d46f"}
Nov 01 12:30:15.469652 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:15.469304    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:30:15.578560 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:15.578459    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2" podStartSLOduration=1.956357046 podStartE2EDuration="16.578446227s" totalImagesPullingTime="14.622089181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:15.577674299 +0000 UTC m=+457.084865538" watchObservedRunningTime="2026-11-01 12:30:15.578446227 +0000 UTC m=+457.085637444"
Nov 01 12:30:24.845762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.845687    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"]
Nov 01 12:30:24.846219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.846103    5725 cpu_manager.go:414] "containerMap: removing container" podUID="d45adcbc-88f1-4a66-8b46-fcbf38ff6d55" containerName="manager"
Nov 01 12:30:24.846219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.846115    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45adcbc-88f1-4a66-8b46-fcbf38ff6d55" containerName="manager"
Nov 01 12:30:24.846219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.846183    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45adcbc-88f1-4a66-8b46-fcbf38ff6d55" containerName="manager"
Nov 01 12:30:24.851295 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.851269    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:30:24.903658 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.903625    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"]
Nov 01 12:30:24.947892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.947853    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 01 12:30:24.947892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.947853    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8fcdm\""
Nov 01 12:30:24.979175 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.979139    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b7c25b-d9f5-4385-8b8e-a82837907434-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9dmlw\" (UID: \"21b7c25b-d9f5-4385-8b8e-a82837907434\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:30:24.979175 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:24.979178    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ldnk\" (UniqueName: \"kubernetes.io/projected/21b7c25b-d9f5-4385-8b8e-a82837907434-kube-api-access-9ldnk\") pod \"llmisvc-controller-manager-869d5dcc5d-9dmlw\" (UID: \"21b7c25b-d9f5-4385-8b8e-a82837907434\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:30:25.058587 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.058552    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-rrdwb"]
Nov 01 12:30:25.062533 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.062509    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.080453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.080424    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b7c25b-d9f5-4385-8b8e-a82837907434-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9dmlw\" (UID: \"21b7c25b-d9f5-4385-8b8e-a82837907434\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:30:25.080593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.080470    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldnk\" (UniqueName: \"kubernetes.io/projected/21b7c25b-d9f5-4385-8b8e-a82837907434-kube-api-access-9ldnk\") pod \"llmisvc-controller-manager-869d5dcc5d-9dmlw\" (UID: \"21b7c25b-d9f5-4385-8b8e-a82837907434\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:30:25.082850 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.082819    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b7c25b-d9f5-4385-8b8e-a82837907434-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9dmlw\" (UID: \"21b7c25b-d9f5-4385-8b8e-a82837907434\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:30:25.115038 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.114923    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-rrdwb"]
Nov 01 12:30:25.115697 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.115675    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jjbhr\""
Nov 01 12:30:25.115814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.115755    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 01 12:30:25.115859 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.115821    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 01 12:30:25.133829 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.133800    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldnk\" (UniqueName: \"kubernetes.io/projected/21b7c25b-d9f5-4385-8b8e-a82837907434-kube-api-access-9ldnk\") pod \"llmisvc-controller-manager-869d5dcc5d-9dmlw\" (UID: \"21b7c25b-d9f5-4385-8b8e-a82837907434\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:30:25.163693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.163658    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:30:25.180992 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.180963    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7234148f-b03b-4e54-8e88-61d41c027559-cert\") pod \"odh-model-controller-769875db56-rrdwb\" (UID: \"7234148f-b03b-4e54-8e88-61d41c027559\") " pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.181158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.181005    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7234148f-b03b-4e54-8e88-61d41c027559-metrics-certs\") pod \"odh-model-controller-769875db56-rrdwb\" (UID: \"7234148f-b03b-4e54-8e88-61d41c027559\") " pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.181158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.181123    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wnnj\" (UniqueName: \"kubernetes.io/projected/7234148f-b03b-4e54-8e88-61d41c027559-kube-api-access-5wnnj\") pod \"odh-model-controller-769875db56-rrdwb\" (UID: \"7234148f-b03b-4e54-8e88-61d41c027559\") " pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.282368 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.282331    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wnnj\" (UniqueName: \"kubernetes.io/projected/7234148f-b03b-4e54-8e88-61d41c027559-kube-api-access-5wnnj\") pod \"odh-model-controller-769875db56-rrdwb\" (UID: \"7234148f-b03b-4e54-8e88-61d41c027559\") " pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.282526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.282477    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7234148f-b03b-4e54-8e88-61d41c027559-cert\") pod \"odh-model-controller-769875db56-rrdwb\" (UID: \"7234148f-b03b-4e54-8e88-61d41c027559\") " pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.282526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.282507    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7234148f-b03b-4e54-8e88-61d41c027559-metrics-certs\") pod \"odh-model-controller-769875db56-rrdwb\" (UID: \"7234148f-b03b-4e54-8e88-61d41c027559\") " pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.284872 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.284843    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7234148f-b03b-4e54-8e88-61d41c027559-cert\") pod \"odh-model-controller-769875db56-rrdwb\" (UID: \"7234148f-b03b-4e54-8e88-61d41c027559\") " pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.285000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.284892    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7234148f-b03b-4e54-8e88-61d41c027559-metrics-certs\") pod \"odh-model-controller-769875db56-rrdwb\" (UID: \"7234148f-b03b-4e54-8e88-61d41c027559\") " pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.336591 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.336558    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wnnj\" (UniqueName: \"kubernetes.io/projected/7234148f-b03b-4e54-8e88-61d41c027559-kube-api-access-5wnnj\") pod \"odh-model-controller-769875db56-rrdwb\" (UID: \"7234148f-b03b-4e54-8e88-61d41c027559\") " pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.375605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.375520    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:25.533639 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.533599    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"]
Nov 01 12:30:25.541808 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:30:25.541775    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21b7c25b_d9f5_4385_8b8e_a82837907434.slice/crio-dbbdda837b1eedb1a2284a066e13768fd3e60a1fec9a896fbd65f96ec9d9770e WatchSource:0}: Error finding container dbbdda837b1eedb1a2284a066e13768fd3e60a1fec9a896fbd65f96ec9d9770e: Status 404 returned error can't find the container with id dbbdda837b1eedb1a2284a066e13768fd3e60a1fec9a896fbd65f96ec9d9770e
Nov 01 12:30:25.745932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:25.745897    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-rrdwb"]
Nov 01 12:30:25.753568 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:30:25.753538    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7234148f_b03b_4e54_8e88_61d41c027559.slice/crio-6d601677e16b3492c3aceb7ab936fc75af50704f9c1b88fa4f6719c5b4fce3a6 WatchSource:0}: Error finding container 6d601677e16b3492c3aceb7ab936fc75af50704f9c1b88fa4f6719c5b4fce3a6: Status 404 returned error can't find the container with id 6d601677e16b3492c3aceb7ab936fc75af50704f9c1b88fa4f6719c5b4fce3a6
Nov 01 12:30:26.484044 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:26.484009    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vfwh2"
Nov 01 12:30:26.535312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:26.535277    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw" event={"ID":"21b7c25b-d9f5-4385-8b8e-a82837907434","Type":"ContainerStarted","Data":"dbbdda837b1eedb1a2284a066e13768fd3e60a1fec9a896fbd65f96ec9d9770e"}
Nov 01 12:30:26.536529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:26.536496    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-rrdwb" event={"ID":"7234148f-b03b-4e54-8e88-61d41c027559","Type":"ContainerStarted","Data":"6d601677e16b3492c3aceb7ab936fc75af50704f9c1b88fa4f6719c5b4fce3a6"}
Nov 01 12:30:28.554351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:28.554262    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw" event={"ID":"21b7c25b-d9f5-4385-8b8e-a82837907434","Type":"ContainerStarted","Data":"6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441"}
Nov 01 12:30:28.554806 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:28.554472    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:30:30.568758 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:30.568716    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-rrdwb" event={"ID":"7234148f-b03b-4e54-8e88-61d41c027559","Type":"ContainerStarted","Data":"a5f5cc0ea1c21c1f6bba89d451508db8f4f3d5da423f81c143b43e704ea5876b"}
Nov 01 12:30:30.569263 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:30.568788    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:30.676126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:30.676039    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-rrdwb" podStartSLOduration=1.8877204060000001 podStartE2EDuration="5.676024277s" totalImagesPullingTime="3.788303871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:30.675375273 +0000 UTC m=+472.182566527" watchObservedRunningTime="2026-11-01 12:30:30.676024277 +0000 UTC m=+472.183215494"
Nov 01 12:30:30.676385 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:30.676365    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw" podStartSLOduration=3.928298673 podStartE2EDuration="6.676360326s" totalImagesPullingTime="2.748061653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:28.709545651 +0000 UTC m=+470.216736878" watchObservedRunningTime="2026-11-01 12:30:30.676360326 +0000 UTC m=+472.183551542"
Nov 01 12:30:31.028116 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:30:31.028032    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:30:41.576874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:41.576841    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-rrdwb"
Nov 01 12:30:41.868761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:41.868723    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-djpcr"]
Nov 01 12:30:41.871967 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:41.871943    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-djpcr"
Nov 01 12:30:41.928513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:41.928481    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-djpcr"]
Nov 01 12:30:41.969472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:41.969434    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mh5\" (UniqueName: \"kubernetes.io/projected/e9c43639-1cac-4db5-ad93-92b9ab53e2b9-kube-api-access-f9mh5\") pod \"authorino-6f6944bcdd-djpcr\" (UID: \"e9c43639-1cac-4db5-ad93-92b9ab53e2b9\") " pod="kuadrant-system/authorino-6f6944bcdd-djpcr"
Nov 01 12:30:42.070409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.070365    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mh5\" (UniqueName: \"kubernetes.io/projected/e9c43639-1cac-4db5-ad93-92b9ab53e2b9-kube-api-access-f9mh5\") pod \"authorino-6f6944bcdd-djpcr\" (UID: \"e9c43639-1cac-4db5-ad93-92b9ab53e2b9\") " pod="kuadrant-system/authorino-6f6944bcdd-djpcr"
Nov 01 12:30:42.124609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.124524    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mh5\" (UniqueName: \"kubernetes.io/projected/e9c43639-1cac-4db5-ad93-92b9ab53e2b9-kube-api-access-f9mh5\") pod \"authorino-6f6944bcdd-djpcr\" (UID: \"e9c43639-1cac-4db5-ad93-92b9ab53e2b9\") " pod="kuadrant-system/authorino-6f6944bcdd-djpcr"
Nov 01 12:30:42.183496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.183447    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-djpcr"
Nov 01 12:30:42.216437 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.216406    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-djpcr"]
Nov 01 12:30:42.246668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.244254    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76cf4d5dd-6z7xm"]
Nov 01 12:30:42.255116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.248694    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm"
Nov 01 12:30:42.271768 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.271726    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v6rc\" (UniqueName: \"kubernetes.io/projected/3dbf55b5-a88b-41f4-96dc-4ec9576ceb00-kube-api-access-5v6rc\") pod \"authorino-76cf4d5dd-6z7xm\" (UID: \"3dbf55b5-a88b-41f4-96dc-4ec9576ceb00\") " pod="kuadrant-system/authorino-76cf4d5dd-6z7xm"
Nov 01 12:30:42.301598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.301568    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76cf4d5dd-6z7xm"]
Nov 01 12:30:42.372733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.372692    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v6rc\" (UniqueName: \"kubernetes.io/projected/3dbf55b5-a88b-41f4-96dc-4ec9576ceb00-kube-api-access-5v6rc\") pod \"authorino-76cf4d5dd-6z7xm\" (UID: \"3dbf55b5-a88b-41f4-96dc-4ec9576ceb00\") " pod="kuadrant-system/authorino-76cf4d5dd-6z7xm"
Nov 01 12:30:42.427209 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.427133    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v6rc\" (UniqueName: \"kubernetes.io/projected/3dbf55b5-a88b-41f4-96dc-4ec9576ceb00-kube-api-access-5v6rc\") pod \"authorino-76cf4d5dd-6z7xm\" (UID: \"3dbf55b5-a88b-41f4-96dc-4ec9576ceb00\") " pod="kuadrant-system/authorino-76cf4d5dd-6z7xm"
Nov 01 12:30:42.553780 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.553734    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"]
Nov 01 12:30:42.558570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.558525    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.564989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.564957    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-djpcr"]
Nov 01 12:30:42.569685 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.569660    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm"
Nov 01 12:30:42.572368 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:30:42.572340    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9c43639_1cac_4db5_ad93_92b9ab53e2b9.slice/crio-304f88450913a69e388c3374dd6e6e5ea2dcf130566e01ebcc131eb006817387 WatchSource:0}: Error finding container 304f88450913a69e388c3374dd6e6e5ea2dcf130566e01ebcc131eb006817387: Status 404 returned error can't find the container with id 304f88450913a69e388c3374dd6e6e5ea2dcf130566e01ebcc131eb006817387
Nov 01 12:30:42.575002 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.574972    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgf87\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-kube-api-access-zgf87\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.575112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.575021    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.575234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.575149    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.575294 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.575233    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.575355 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.575299    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.575410 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.575383    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.575463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.575423    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.575463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.575448    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.575463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.575479    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.648683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.646000    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"]
Nov 01 12:30:42.655212 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.655181    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-djpcr" event={"ID":"e9c43639-1cac-4db5-ad93-92b9ab53e2b9","Type":"ContainerStarted","Data":"304f88450913a69e388c3374dd6e6e5ea2dcf130566e01ebcc131eb006817387"}
Nov 01 12:30:42.676573 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.676534    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.676761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.676598    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf87\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-kube-api-access-zgf87\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.676761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.676630    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.676761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.676694    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.676761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.676736    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.676976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.676758    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.676976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.676878    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.677513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.677462    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.677513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.677493    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.677513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.677102    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.677704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.677403    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.677704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.677153    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.677704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.677251    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.678224 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.678201    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.679910 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.679882    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.680190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.680170    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.735618 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.735579    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.735835 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.735806    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76cf4d5dd-6z7xm"]
Nov 01 12:30:42.736421 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.736393    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf87\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-kube-api-access-zgf87\") pod \"maas-default-gateway-openshift-default-7df854bf67-4ns6p\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.757990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.757957    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f45579c86-g6wk7"]
Nov 01 12:30:42.761519 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.761491    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:30:42.778379 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.778349    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89618e46-2fde-41aa-9eea-6b4fb6fe2413-tls-cert\") pod \"authorino-5f45579c86-g6wk7\" (UID: \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\") " pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:30:42.778540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.778394    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6lh\" (UniqueName: \"kubernetes.io/projected/89618e46-2fde-41aa-9eea-6b4fb6fe2413-kube-api-access-cf6lh\") pod \"authorino-5f45579c86-g6wk7\" (UID: \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\") " pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:30:42.811530 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.811448    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 12:30:42.814957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.814926    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f45579c86-g6wk7"]
Nov 01 12:30:42.877021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.876985    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:42.879112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.878912    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89618e46-2fde-41aa-9eea-6b4fb6fe2413-tls-cert\") pod \"authorino-5f45579c86-g6wk7\" (UID: \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\") " pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:30:42.879112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.878954    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6lh\" (UniqueName: \"kubernetes.io/projected/89618e46-2fde-41aa-9eea-6b4fb6fe2413-kube-api-access-cf6lh\") pod \"authorino-5f45579c86-g6wk7\" (UID: \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\") " pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:30:42.881589 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.881562    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89618e46-2fde-41aa-9eea-6b4fb6fe2413-tls-cert\") pod \"authorino-5f45579c86-g6wk7\" (UID: \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\") " pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:30:42.934735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.934700    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6lh\" (UniqueName: \"kubernetes.io/projected/89618e46-2fde-41aa-9eea-6b4fb6fe2413-kube-api-access-cf6lh\") pod \"authorino-5f45579c86-g6wk7\" (UID: \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\") " pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:30:42.951560 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:42.951522    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76cf4d5dd-6z7xm"]
Nov 01 12:30:42.958060 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:30:42.958020    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dbf55b5_a88b_41f4_96dc_4ec9576ceb00.slice/crio-90d2679f4546e8a8fc6b9defe22ad7cad28f4f2a58cd17c1273b373fed089e3f WatchSource:0}: Error finding container 90d2679f4546e8a8fc6b9defe22ad7cad28f4f2a58cd17c1273b373fed089e3f: Status 404 returned error can't find the container with id 90d2679f4546e8a8fc6b9defe22ad7cad28f4f2a58cd17c1273b373fed089e3f
Nov 01 12:30:43.074096 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.074060    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:30:43.258781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.258747    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"]
Nov 01 12:30:43.266687 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:30:43.266657    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ccefa7_cab2_4c4d_951e_1036ecaff01b.slice/crio-7040c21842db367ff9f4a06841f6610a78ea065fb78cb1a9740f34391c205bd6 WatchSource:0}: Error finding container 7040c21842db367ff9f4a06841f6610a78ea065fb78cb1a9740f34391c205bd6: Status 404 returned error can't find the container with id 7040c21842db367ff9f4a06841f6610a78ea065fb78cb1a9740f34391c205bd6
Nov 01 12:30:43.269391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.269350    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:30:43.269469 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.269439    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:30:43.269508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.269485    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:30:43.444542 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.444509    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f45579c86-g6wk7"]
Nov 01 12:30:43.452240 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:30:43.452204    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89618e46_2fde_41aa_9eea_6b4fb6fe2413.slice/crio-7f63854148cf0e428cf753877dfeb265e521de1c728ad1feac2354d26a712702 WatchSource:0}: Error finding container 7f63854148cf0e428cf753877dfeb265e521de1c728ad1feac2354d26a712702: Status 404 returned error can't find the container with id 7f63854148cf0e428cf753877dfeb265e521de1c728ad1feac2354d26a712702
Nov 01 12:30:43.663407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.663315    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" event={"ID":"91ccefa7-cab2-4c4d-951e-1036ecaff01b","Type":"ContainerStarted","Data":"856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589"}
Nov 01 12:30:43.663407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.663383    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" event={"ID":"91ccefa7-cab2-4c4d-951e-1036ecaff01b","Type":"ContainerStarted","Data":"7040c21842db367ff9f4a06841f6610a78ea065fb78cb1a9740f34391c205bd6"}
Nov 01 12:30:43.664702 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.664676    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f45579c86-g6wk7" event={"ID":"89618e46-2fde-41aa-9eea-6b4fb6fe2413","Type":"ContainerStarted","Data":"7f63854148cf0e428cf753877dfeb265e521de1c728ad1feac2354d26a712702"}
Nov 01 12:30:43.665827 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.665800    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm" event={"ID":"3dbf55b5-a88b-41f4-96dc-4ec9576ceb00","Type":"ContainerStarted","Data":"90d2679f4546e8a8fc6b9defe22ad7cad28f4f2a58cd17c1273b373fed089e3f"}
Nov 01 12:30:43.778862 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.777938    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" podStartSLOduration=1.777919689 podStartE2EDuration="1.777919689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:43.777458342 +0000 UTC m=+485.284649598" watchObservedRunningTime="2026-11-01 12:30:43.777919689 +0000 UTC m=+485.285110907"
Nov 01 12:30:43.877755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:43.877719    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:44.673069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:44.673013    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm" event={"ID":"3dbf55b5-a88b-41f4-96dc-4ec9576ceb00","Type":"ContainerStarted","Data":"98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be"}
Nov 01 12:30:44.673069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:44.673044    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm" podUID="3dbf55b5-a88b-41f4-96dc-4ec9576ceb00" containerName="authorino" containerID="cri-o://98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be" gracePeriod=30
Nov 01 12:30:44.674727 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:44.674701    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-djpcr" event={"ID":"e9c43639-1cac-4db5-ad93-92b9ab53e2b9","Type":"ContainerStarted","Data":"731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d"}
Nov 01 12:30:44.674837 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:44.674767    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-djpcr" podUID="e9c43639-1cac-4db5-ad93-92b9ab53e2b9" containerName="authorino" containerID="cri-o://731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d" gracePeriod=30
Nov 01 12:30:44.880571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:44.880536    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4ns6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:30:44.880721 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:44.880603    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 12:30:44.896421 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:44.896091    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-djpcr" podStartSLOduration=2.265124668 podStartE2EDuration="3.896037628s" totalImagesPullingTime="1.63091296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:44.894404849 +0000 UTC m=+486.401596066" watchObservedRunningTime="2026-11-01 12:30:44.896037628 +0000 UTC m=+486.403228845"
Nov 01 12:30:44.899016 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:44.898500    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm" podStartSLOduration=1.491726761 podStartE2EDuration="2.898486457s" totalImagesPullingTime="1.406759696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:44.782208822 +0000 UTC m=+486.289400041" watchObservedRunningTime="2026-11-01 12:30:44.898486457 +0000 UTC m=+486.405677675"
Nov 01 12:30:45.023948 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.023918    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm"
Nov 01 12:30:45.057444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.057236    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-djpcr"
Nov 01 12:30:45.101356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.101318    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c43639-1cac-4db5-ad93-92b9ab53e2b9-kube-api-access-f9mh5\" (UniqueName: \"kubernetes.io/projected/e9c43639-1cac-4db5-ad93-92b9ab53e2b9-kube-api-access-f9mh5\") pod \"e9c43639-1cac-4db5-ad93-92b9ab53e2b9\" (UID: \"e9c43639-1cac-4db5-ad93-92b9ab53e2b9\") "
Nov 01 12:30:45.101550 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.101510    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbf55b5-a88b-41f4-96dc-4ec9576ceb00-kube-api-access-5v6rc\" (UniqueName: \"kubernetes.io/projected/3dbf55b5-a88b-41f4-96dc-4ec9576ceb00-kube-api-access-5v6rc\") pod \"3dbf55b5-a88b-41f4-96dc-4ec9576ceb00\" (UID: \"3dbf55b5-a88b-41f4-96dc-4ec9576ceb00\") "
Nov 01 12:30:45.103468 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.103441    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbf55b5-a88b-41f4-96dc-4ec9576ceb00-kube-api-access-5v6rc" pod "3dbf55b5-a88b-41f4-96dc-4ec9576ceb00" (UID: "3dbf55b5-a88b-41f4-96dc-4ec9576ceb00"). InnerVolumeSpecName "kube-api-access-5v6rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:30:45.103623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.103474    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c43639-1cac-4db5-ad93-92b9ab53e2b9-kube-api-access-f9mh5" pod "e9c43639-1cac-4db5-ad93-92b9ab53e2b9" (UID: "e9c43639-1cac-4db5-ad93-92b9ab53e2b9"). InnerVolumeSpecName "kube-api-access-f9mh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:30:45.203041 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.203003    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9mh5\" (UniqueName: \"kubernetes.io/projected/e9c43639-1cac-4db5-ad93-92b9ab53e2b9-kube-api-access-f9mh5\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:30:45.203041 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.203035    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v6rc\" (UniqueName: \"kubernetes.io/projected/3dbf55b5-a88b-41f4-96dc-4ec9576ceb00-kube-api-access-5v6rc\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:30:45.683989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.683892    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f45579c86-g6wk7" event={"ID":"89618e46-2fde-41aa-9eea-6b4fb6fe2413","Type":"ContainerStarted","Data":"5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d"}
Nov 01 12:30:45.685778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.685734    5725 generic.go:356] "Generic (PLEG): container finished" podID="3dbf55b5-a88b-41f4-96dc-4ec9576ceb00" containerID="98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be" exitCode=0
Nov 01 12:30:45.685916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.685835    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm"
Nov 01 12:30:45.685916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.685868    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm" event={"ID":"3dbf55b5-a88b-41f4-96dc-4ec9576ceb00","Type":"ContainerDied","Data":"98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be"}
Nov 01 12:30:45.686031 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.685920    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76cf4d5dd-6z7xm" event={"ID":"3dbf55b5-a88b-41f4-96dc-4ec9576ceb00","Type":"ContainerDied","Data":"90d2679f4546e8a8fc6b9defe22ad7cad28f4f2a58cd17c1273b373fed089e3f"}
Nov 01 12:30:45.686031 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.685945    5725 scope.go:122] "RemoveContainer" containerID="98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be"
Nov 01 12:30:45.688035 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.687998    5725 generic.go:356] "Generic (PLEG): container finished" podID="e9c43639-1cac-4db5-ad93-92b9ab53e2b9" containerID="731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d" exitCode=0
Nov 01 12:30:45.688216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.688111    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-djpcr"
Nov 01 12:30:45.688216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.688118    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-djpcr" event={"ID":"e9c43639-1cac-4db5-ad93-92b9ab53e2b9","Type":"ContainerDied","Data":"731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d"}
Nov 01 12:30:45.688216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.688149    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-djpcr" event={"ID":"e9c43639-1cac-4db5-ad93-92b9ab53e2b9","Type":"ContainerDied","Data":"304f88450913a69e388c3374dd6e6e5ea2dcf130566e01ebcc131eb006817387"}
Nov 01 12:30:45.702330 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.702300    5725 generic.go:356] "Generic (PLEG): container finished" podID="9ec5523e-d2ad-414c-9186-003789159540" containerID="f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339" exitCode=0
Nov 01 12:30:45.702466 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.702349    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" event={"ID":"9ec5523e-d2ad-414c-9186-003789159540","Type":"ContainerDied","Data":"f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339"}
Nov 01 12:30:45.703144 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.703118    5725 scope.go:122] "RemoveContainer" containerID="f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339"
Nov 01 12:30:45.710150 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.710105    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:30:45.710150 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.710138    5725 scope.go:122] "RemoveContainer" containerID="98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be"
Nov 01 12:30:45.710314 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.710206    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:30:45.710314 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.710253    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:30:45.710488 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:30:45.710467    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be\": container with ID starting with 98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be not found: ID does not exist" containerID="98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be"
Nov 01 12:30:45.710555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.710500    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be"} err="failed to get container status \"98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be\": rpc error: code = NotFound desc = could not find container \"98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be\": container with ID starting with 98250c9c3077a65f70717d2c1d5b788fab9c3539de6f2d94903b29826d45e6be not found: ID does not exist"
Nov 01 12:30:45.710555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.710525    5725 scope.go:122] "RemoveContainer" containerID="731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d"
Nov 01 12:30:45.726138 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.726117    5725 scope.go:122] "RemoveContainer" containerID="731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d"
Nov 01 12:30:45.726593 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:30:45.726503    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d\": container with ID starting with 731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d not found: ID does not exist" containerID="731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d"
Nov 01 12:30:45.726593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.726543    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d"} err="failed to get container status \"731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d\": rpc error: code = NotFound desc = could not find container \"731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d\": container with ID starting with 731700d81295b67ea8aeaf3b3f11f815a44380708b219e323bc73a08d002237d not found: ID does not exist"
Nov 01 12:30:45.801409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.801307    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f45579c86-g6wk7" podStartSLOduration=2.442973973 podStartE2EDuration="3.8012855s" totalImagesPullingTime="1.358311527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:45.799347605 +0000 UTC m=+487.306538824" watchObservedRunningTime="2026-11-01 12:30:45.8012855 +0000 UTC m=+487.308476718"
Nov 01 12:30:45.868191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.853833    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2tmwx"]
Nov 01 12:30:45.868191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.859193    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx" podUID="3bd9b065-d579-48ad-890a-61a43423e20f" containerName="authorino" containerID="cri-o://062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1" gracePeriod=30
Nov 01 12:30:45.879223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.879179    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4ns6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:30:45.879789 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.879268    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 12:30:45.913967 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.913887    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-djpcr"]
Nov 01 12:30:45.916251 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:45.916221    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-djpcr"]
Nov 01 12:30:46.155567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.155532    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76cf4d5dd-6z7xm"]
Nov 01 12:30:46.168940 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.168903    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76cf4d5dd-6z7xm"]
Nov 01 12:30:46.281974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.281948    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx"
Nov 01 12:30:46.316544 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.316513    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd9b065-d579-48ad-890a-61a43423e20f-kube-api-access-l4c5z\" (UniqueName: \"kubernetes.io/projected/3bd9b065-d579-48ad-890a-61a43423e20f-kube-api-access-l4c5z\") pod \"3bd9b065-d579-48ad-890a-61a43423e20f\" (UID: \"3bd9b065-d579-48ad-890a-61a43423e20f\") "
Nov 01 12:30:46.318682 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.318654    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd9b065-d579-48ad-890a-61a43423e20f-kube-api-access-l4c5z" pod "3bd9b065-d579-48ad-890a-61a43423e20f" (UID: "3bd9b065-d579-48ad-890a-61a43423e20f"). InnerVolumeSpecName "kube-api-access-l4c5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:30:46.422641 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.422596    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4c5z\" (UniqueName: \"kubernetes.io/projected/3bd9b065-d579-48ad-890a-61a43423e20f-kube-api-access-l4c5z\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:30:46.615182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.615105    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbf55b5-a88b-41f4-96dc-4ec9576ceb00" path="/var/lib/kubelet/pods/3dbf55b5-a88b-41f4-96dc-4ec9576ceb00/volumes"
Nov 01 12:30:46.615537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.615518    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c43639-1cac-4db5-ad93-92b9ab53e2b9" path="/var/lib/kubelet/pods/e9c43639-1cac-4db5-ad93-92b9ab53e2b9/volumes"
Nov 01 12:30:46.710900 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.710866    5725 generic.go:356] "Generic (PLEG): container finished" podID="3bd9b065-d579-48ad-890a-61a43423e20f" containerID="062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1" exitCode=0
Nov 01 12:30:46.711316 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.710912    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx"
Nov 01 12:30:46.711316 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.710950    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx" event={"ID":"3bd9b065-d579-48ad-890a-61a43423e20f","Type":"ContainerDied","Data":"062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1"}
Nov 01 12:30:46.711316 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.711203    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2tmwx" event={"ID":"3bd9b065-d579-48ad-890a-61a43423e20f","Type":"ContainerDied","Data":"6f1801fa4896739f7aea1eb6ef9a0e36bbaf70c1714c516bab9f67f17d7a72ad"}
Nov 01 12:30:46.711316 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.711259    5725 scope.go:122] "RemoveContainer" containerID="062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1"
Nov 01 12:30:46.714753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.714726    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" event={"ID":"9ec5523e-d2ad-414c-9186-003789159540","Type":"ContainerStarted","Data":"42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732"}
Nov 01 12:30:46.720953 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.720938    5725 scope.go:122] "RemoveContainer" containerID="062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1"
Nov 01 12:30:46.721241 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:30:46.721225    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1\": container with ID starting with 062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1 not found: ID does not exist" containerID="062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1"
Nov 01 12:30:46.721285 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.721250    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1"} err="failed to get container status \"062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1\": rpc error: code = NotFound desc = could not find container \"062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1\": container with ID starting with 062ab23b34cee10f9b0be6abf1c15003778657df8385eab498ad72207b7f4ff1 not found: ID does not exist"
Nov 01 12:30:46.882778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.882252    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4ns6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:30:46.882778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:46.882317    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 12:30:47.001745 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:47.001704    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2tmwx"]
Nov 01 12:30:47.007462 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:47.007434    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2tmwx"]
Nov 01 12:30:47.361621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:47.361586    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:30:47.361994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:47.361969    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-h6tl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:30:47.362181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:47.362154    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 12:30:47.501105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:47.501075    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:30:47.877734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:47.877403    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4ns6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:30:47.877734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:47.877454    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 12:30:48.364075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:48.363568    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-h6tl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:30:48.364075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:48.363631    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 12:30:48.615663 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:48.615584    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd9b065-d579-48ad-890a-61a43423e20f" path="/var/lib/kubelet/pods/3bd9b065-d579-48ad-890a-61a43423e20f/volumes"
Nov 01 12:30:48.878513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:48.878433    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4ns6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:30:48.878991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:48.878683    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 01 12:30:49.360728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:49.360685    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-h6tl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:30:49.360913 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:49.360758    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 12:30:49.881691 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:49.881654    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:49.882220 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:49.882040    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:49.882690 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:49.882674    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:30:50.361253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:50.361218    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-h6tl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:30:50.361442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:50.361276    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 12:30:51.360371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:51.360320    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-h6tl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:30:51.360893 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:51.360403    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 12:30:52.364550 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:52.364521    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:30:52.365578 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:52.365559    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:30:59.563215 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:30:59.563183    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:31:01.026413 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:31:01.026354    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:31:12.768692 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.768658    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"]
Nov 01 12:31:12.769141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.769105    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3bd9b065-d579-48ad-890a-61a43423e20f" containerName="authorino"
Nov 01 12:31:12.769141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.769117    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd9b065-d579-48ad-890a-61a43423e20f" containerName="authorino"
Nov 01 12:31:12.769141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.769134    5725 cpu_manager.go:414] "containerMap: removing container" podUID="e9c43639-1cac-4db5-ad93-92b9ab53e2b9" containerName="authorino"
Nov 01 12:31:12.769141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.769140    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c43639-1cac-4db5-ad93-92b9ab53e2b9" containerName="authorino"
Nov 01 12:31:12.769271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.769157    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3dbf55b5-a88b-41f4-96dc-4ec9576ceb00" containerName="authorino"
Nov 01 12:31:12.769271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.769162    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbf55b5-a88b-41f4-96dc-4ec9576ceb00" containerName="authorino"
Nov 01 12:31:12.769271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.769226    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd9b065-d579-48ad-890a-61a43423e20f" containerName="authorino"
Nov 01 12:31:12.769271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.769234    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c43639-1cac-4db5-ad93-92b9ab53e2b9" containerName="authorino"
Nov 01 12:31:12.769271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.769242    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbf55b5-a88b-41f4-96dc-4ec9576ceb00" containerName="authorino"
Nov 01 12:31:12.773776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.773756    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.911391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.911359    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.911561 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.911413    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.911561 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.911434    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.911561 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.911524    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznvt\" (UniqueName: \"kubernetes.io/projected/fbe0ab95-980d-496a-8437-b3c6e552f9bc-kube-api-access-vznvt\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.911668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.911562    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.911668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.911579    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.911668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.911649    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.911767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.911697    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.911767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.911755    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:12.956714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:12.956675    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"]
Nov 01 12:31:13.012366 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012333    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.012570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012375    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.012570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012427    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.012570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012473    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.012570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012525    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.012570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012554    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.012836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012584    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznvt\" (UniqueName: \"kubernetes.io/projected/fbe0ab95-980d-496a-8437-b3c6e552f9bc-kube-api-access-vznvt\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.012836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012621    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.012836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012646    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.012999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012980    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.013092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.012980    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.013151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.013101    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.013270 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.013245    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.013419 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.013399    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.014909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.014884    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.015006 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.014919    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.066096 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.066008    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe0ab95-980d-496a-8437-b3c6e552f9bc-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.066257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.066234    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznvt\" (UniqueName: \"kubernetes.io/projected/fbe0ab95-980d-496a-8437-b3c6e552f9bc-kube-api-access-vznvt\") pod \"maas-default-gateway-openshift-default-7df854bf67-hs25d\" (UID: \"fbe0ab95-980d-496a-8437-b3c6e552f9bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.085937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.085901    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:13.510958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.510910    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"]
Nov 01 12:31:13.519523 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:31:13.519490    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe0ab95_980d_496a_8437_b3c6e552f9bc.slice/crio-47b39bc932be602dc6b3248a14035eaa9d92e5074c6073ff044e3aac47d7df77 WatchSource:0}: Error finding container 47b39bc932be602dc6b3248a14035eaa9d92e5074c6073ff044e3aac47d7df77: Status 404 returned error can't find the container with id 47b39bc932be602dc6b3248a14035eaa9d92e5074c6073ff044e3aac47d7df77
Nov 01 12:31:13.522678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.522643    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:31:13.522752 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.522739    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:31:13.522795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.522768    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:31:13.898932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.898828    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" event={"ID":"fbe0ab95-980d-496a-8437-b3c6e552f9bc","Type":"ContainerStarted","Data":"9d138924591fda1aee414927746e708aaaba928a0be59f071a07d2fce7432121"}
Nov 01 12:31:13.898932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:13.898894    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" event={"ID":"fbe0ab95-980d-496a-8437-b3c6e552f9bc","Type":"ContainerStarted","Data":"47b39bc932be602dc6b3248a14035eaa9d92e5074c6073ff044e3aac47d7df77"}
Nov 01 12:31:14.059174 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:14.059120    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podStartSLOduration=2.059105149 podStartE2EDuration="2.059105149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:31:14.056646359 +0000 UTC m=+515.563837575" watchObservedRunningTime="2026-11-01 12:31:14.059105149 +0000 UTC m=+515.566296365"
Nov 01 12:31:14.086168 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:14.086137    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:14.087532 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:14.087504    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 12:31:14.087659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:14.087551    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 12:31:15.086896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:15.086860    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 12:31:15.087332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:15.086918    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 12:31:16.086746 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:16.086698    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 12:31:16.086923 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:16.086767    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 12:31:17.086442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:17.086403    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 12:31:17.086611 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:17.086459    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 12:31:18.087208 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:18.087170    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 12:31:18.087606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:18.087226    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 12:31:19.086766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:19.086727    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 12:31:19.086950 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:19.086779    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 12:31:20.090978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:20.090946    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:20.091376 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:20.091260    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:20.091916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:20.091898    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d"
Nov 01 12:31:27.876429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.876392    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"]
Nov 01 12:31:27.880033 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.880012    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:27.968191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.968153    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"]
Nov 01 12:31:27.975440 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.975414    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwfq\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-kube-api-access-hjwfq\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:27.975598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.975528    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:27.975598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.975586    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:27.975698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.975625    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:27.975698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.975678    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a99f825-ca55-4acf-97e4-10ea52ae71da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:27.975811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.975711    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:27.975811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.975761    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:27.975811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.975794    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:27.975932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:27.975826    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.076799 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.076766    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.076951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.076823    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwfq\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-kube-api-access-hjwfq\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.076951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.076894    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.076951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.076929    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.076969    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.077113    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a99f825-ca55-4acf-97e4-10ea52ae71da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.077171    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.077206    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.077245    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.077289    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.077311    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.077546    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077641 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.077580    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.077929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.077905    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a99f825-ca55-4acf-97e4-10ea52ae71da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.079436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.079410    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.079436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.079430    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.134133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.134038    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.134288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.134240    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwfq\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-kube-api-access-hjwfq\") pod \"maas-default-gateway-openshift-default-7df854bf67-znm24\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.194277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.194242    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:28.623110 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.623074    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"]
Nov 01 12:31:28.630176 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:31:28.630145    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a99f825_ca55_4acf_97e4_10ea52ae71da.slice/crio-9fecaffddd20be1bba913e8baa979a114c688223e2729851c11514399c49e1ec WatchSource:0}: Error finding container 9fecaffddd20be1bba913e8baa979a114c688223e2729851c11514399c49e1ec: Status 404 returned error can't find the container with id 9fecaffddd20be1bba913e8baa979a114c688223e2729851c11514399c49e1ec
Nov 01 12:31:28.632435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.632403    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:31:28.632522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.632470    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:31:28.632522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.632500    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:31:28.984543 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.984502    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" event={"ID":"4a99f825-ca55-4acf-97e4-10ea52ae71da","Type":"ContainerStarted","Data":"7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9"}
Nov 01 12:31:28.985101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:28.984565    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" event={"ID":"4a99f825-ca55-4acf-97e4-10ea52ae71da","Type":"ContainerStarted","Data":"9fecaffddd20be1bba913e8baa979a114c688223e2729851c11514399c49e1ec"}
Nov 01 12:31:29.099032 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:29.098982    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" podStartSLOduration=2.098966997 podStartE2EDuration="2.098966997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:31:29.097194882 +0000 UTC m=+530.604386098" watchObservedRunningTime="2026-11-01 12:31:29.098966997 +0000 UTC m=+530.606158208"
Nov 01 12:31:29.194875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:29.194843    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:29.196470 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:29.196439    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-znm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 12:31:29.196585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:29.196490    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 12:31:30.195354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:30.195316    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-znm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 12:31:30.195758 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:30.195371    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 12:31:31.044113 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:31:31.027125    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:31:31.195143 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:31.195111    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-znm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 12:31:31.195342 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:31.195169    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 12:31:32.195272 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:32.195233    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-znm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 12:31:32.195686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:32.195300    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 12:31:33.194788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:33.194753    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-znm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 12:31:33.194974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:33.194806    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 12:31:34.195449 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:34.195405    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-znm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 12:31:34.195843 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:34.195478    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 12:31:35.199155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:35.199120    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:35.199537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:35.199419    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:35.200141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:35.200120    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:31:47.261871 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.261835    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"]
Nov 01 12:31:47.267321 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.267297    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:31:47.318488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.318450    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"]
Nov 01 12:31:47.381854 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.381816    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2622\" (UniqueName: \"kubernetes.io/projected/5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa-kube-api-access-p2622\") pod \"llmisvc-controller-manager-6466c554d8-dbmkl\" (UID: \"5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:31:47.382080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.381893    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa-cert\") pod \"llmisvc-controller-manager-6466c554d8-dbmkl\" (UID: \"5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:31:47.482668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.482629    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa-cert\") pod \"llmisvc-controller-manager-6466c554d8-dbmkl\" (UID: \"5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:31:47.482862 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.482750    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2622\" (UniqueName: \"kubernetes.io/projected/5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa-kube-api-access-p2622\") pod \"llmisvc-controller-manager-6466c554d8-dbmkl\" (UID: \"5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:31:47.485116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.485090    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa-cert\") pod \"llmisvc-controller-manager-6466c554d8-dbmkl\" (UID: \"5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:31:47.537084 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.536990    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2622\" (UniqueName: \"kubernetes.io/projected/5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa-kube-api-access-p2622\") pod \"llmisvc-controller-manager-6466c554d8-dbmkl\" (UID: \"5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:31:47.580877 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.580843    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:31:47.951131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:47.951045    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"]
Nov 01 12:31:47.958550 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:31:47.958522    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdd86a2_a4f9_4786_99e9_8d7d5e6bfdaa.slice/crio-a10f7f03ce6845a305c9953e807ba3408664ee0f59d6fe6a104f541b1db65e6a WatchSource:0}: Error finding container a10f7f03ce6845a305c9953e807ba3408664ee0f59d6fe6a104f541b1db65e6a: Status 404 returned error can't find the container with id a10f7f03ce6845a305c9953e807ba3408664ee0f59d6fe6a104f541b1db65e6a
Nov 01 12:31:48.103124 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:48.103079    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl" event={"ID":"5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa","Type":"ContainerStarted","Data":"a10f7f03ce6845a305c9953e807ba3408664ee0f59d6fe6a104f541b1db65e6a"}
Nov 01 12:31:49.110702 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:49.110614    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl" event={"ID":"5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa","Type":"ContainerStarted","Data":"f9ca79b35aef3a669039864fcf2cd0df5e6f2e25b906f8a65f135e5ea1ba3f4c"}
Nov 01 12:31:49.111128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:49.110745    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:31:49.221277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:31:49.221213    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl" podStartSLOduration=1.4247015219999999 podStartE2EDuration="2.221191878s" totalImagesPullingTime="796.490356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:31:49.219043324 +0000 UTC m=+550.726234559" watchObservedRunningTime="2026-11-01 12:31:49.221191878 +0000 UTC m=+550.728383099"
Nov 01 12:32:01.030088 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:32:01.030021    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:32:20.118556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.118523    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbmkl"
Nov 01 12:32:20.284913 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.284869    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"]
Nov 01 12:32:20.285223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.285174    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw" podUID="21b7c25b-d9f5-4385-8b8e-a82837907434" containerName="manager" containerID="cri-o://6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441" gracePeriod=30
Nov 01 12:32:20.546714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.546689    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:32:20.691189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.691094    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b7c25b-d9f5-4385-8b8e-a82837907434-kube-api-access-9ldnk\" (UniqueName: \"kubernetes.io/projected/21b7c25b-d9f5-4385-8b8e-a82837907434-kube-api-access-9ldnk\") pod \"21b7c25b-d9f5-4385-8b8e-a82837907434\" (UID: \"21b7c25b-d9f5-4385-8b8e-a82837907434\") "
Nov 01 12:32:20.691359 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.691195    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b7c25b-d9f5-4385-8b8e-a82837907434-cert\" (UniqueName: \"kubernetes.io/secret/21b7c25b-d9f5-4385-8b8e-a82837907434-cert\") pod \"21b7c25b-d9f5-4385-8b8e-a82837907434\" (UID: \"21b7c25b-d9f5-4385-8b8e-a82837907434\") "
Nov 01 12:32:20.693227 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.693201    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b7c25b-d9f5-4385-8b8e-a82837907434-cert" pod "21b7c25b-d9f5-4385-8b8e-a82837907434" (UID: "21b7c25b-d9f5-4385-8b8e-a82837907434"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:32:20.693345 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.693300    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b7c25b-d9f5-4385-8b8e-a82837907434-kube-api-access-9ldnk" pod "21b7c25b-d9f5-4385-8b8e-a82837907434" (UID: "21b7c25b-d9f5-4385-8b8e-a82837907434"). InnerVolumeSpecName "kube-api-access-9ldnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:32:20.796932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.796898    5725 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b7c25b-d9f5-4385-8b8e-a82837907434-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:32:20.797127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:20.796944    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ldnk\" (UniqueName: \"kubernetes.io/projected/21b7c25b-d9f5-4385-8b8e-a82837907434-kube-api-access-9ldnk\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:32:21.297961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:21.297924    5725 generic.go:356] "Generic (PLEG): container finished" podID="21b7c25b-d9f5-4385-8b8e-a82837907434" containerID="6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441" exitCode=0
Nov 01 12:32:21.298466 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:21.297980    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw" event={"ID":"21b7c25b-d9f5-4385-8b8e-a82837907434","Type":"ContainerDied","Data":"6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441"}
Nov 01 12:32:21.298466 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:21.297994    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"
Nov 01 12:32:21.298466 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:21.298062    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw" event={"ID":"21b7c25b-d9f5-4385-8b8e-a82837907434","Type":"ContainerDied","Data":"dbbdda837b1eedb1a2284a066e13768fd3e60a1fec9a896fbd65f96ec9d9770e"}
Nov 01 12:32:21.298466 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:21.298096    5725 scope.go:122] "RemoveContainer" containerID="6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441"
Nov 01 12:32:21.308333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:21.308317    5725 scope.go:122] "RemoveContainer" containerID="6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441"
Nov 01 12:32:21.308603 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:32:21.308586    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441\": container with ID starting with 6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441 not found: ID does not exist" containerID="6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441"
Nov 01 12:32:21.308652 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:21.308610    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441"} err="failed to get container status \"6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441\": rpc error: code = NotFound desc = could not find container \"6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441\": container with ID starting with 6bcc106478461a1ae6063ad0797b79cc83b8432bdb4b539b1d78cb4c60ed0441 not found: ID does not exist"
Nov 01 12:32:21.455186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:21.455148    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"]
Nov 01 12:32:21.501944 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:21.501912    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9dmlw"]
Nov 01 12:32:22.616126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:22.616095    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b7c25b-d9f5-4385-8b8e-a82837907434" path="/var/lib/kubelet/pods/21b7c25b-d9f5-4385-8b8e-a82837907434/volumes"
Nov 01 12:32:25.323581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:25.323549    5725 generic.go:356] "Generic (PLEG): container finished" podID="a52ae595-903b-4c45-9802-e106d8519f07" containerID="72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38" exitCode=1
Nov 01 12:32:25.323988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:25.323600    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mbkth" event={"ID":"a52ae595-903b-4c45-9802-e106d8519f07","Type":"ContainerDied","Data":"72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38"}
Nov 01 12:32:25.323988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:25.323961    5725 scope.go:122] "RemoveContainer" containerID="72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38"
Nov 01 12:32:27.337435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:27.337399    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mbkth" event={"ID":"a52ae595-903b-4c45-9802-e106d8519f07","Type":"ContainerStarted","Data":"5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3"}
Nov 01 12:32:27.337822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:27.337641    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:32:31.026959 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:32:31.026915    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:32:38.346102 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:38.346069    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 12:32:38.773180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:38.773146    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:32:38.776235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:38.776208    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:32:38.780709 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:38.780688    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:32:38.783621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:32:38.783602    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:33:01.031461 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:33:01.031415    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:33:31.031167 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:33:31.031076    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:33:41.598237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.598163    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f9995dd6f-pppw8"]
Nov 01 12:33:41.599324 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.599298    5725 cpu_manager.go:414] "containerMap: removing container" podUID="21b7c25b-d9f5-4385-8b8e-a82837907434" containerName="manager"
Nov 01 12:33:41.599324 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.599321    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b7c25b-d9f5-4385-8b8e-a82837907434" containerName="manager"
Nov 01 12:33:41.599478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.599401    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b7c25b-d9f5-4385-8b8e-a82837907434" containerName="manager"
Nov 01 12:33:41.602605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.602589    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9995dd6f-pppw8"
Nov 01 12:33:41.652748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.652715    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f9995dd6f-pppw8"]
Nov 01 12:33:41.671359 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.671320    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a563ddad-4fab-4b9f-9e01-c325efed519a-tls-cert\") pod \"authorino-5f9995dd6f-pppw8\" (UID: \"a563ddad-4fab-4b9f-9e01-c325efed519a\") " pod="kuadrant-system/authorino-5f9995dd6f-pppw8"
Nov 01 12:33:41.671532 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.671375    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctqc\" (UniqueName: \"kubernetes.io/projected/a563ddad-4fab-4b9f-9e01-c325efed519a-kube-api-access-sctqc\") pod \"authorino-5f9995dd6f-pppw8\" (UID: \"a563ddad-4fab-4b9f-9e01-c325efed519a\") " pod="kuadrant-system/authorino-5f9995dd6f-pppw8"
Nov 01 12:33:41.772383 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.772342    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctqc\" (UniqueName: \"kubernetes.io/projected/a563ddad-4fab-4b9f-9e01-c325efed519a-kube-api-access-sctqc\") pod \"authorino-5f9995dd6f-pppw8\" (UID: \"a563ddad-4fab-4b9f-9e01-c325efed519a\") " pod="kuadrant-system/authorino-5f9995dd6f-pppw8"
Nov 01 12:33:41.772581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.772483    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a563ddad-4fab-4b9f-9e01-c325efed519a-tls-cert\") pod \"authorino-5f9995dd6f-pppw8\" (UID: \"a563ddad-4fab-4b9f-9e01-c325efed519a\") " pod="kuadrant-system/authorino-5f9995dd6f-pppw8"
Nov 01 12:33:41.775042 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.775021    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a563ddad-4fab-4b9f-9e01-c325efed519a-tls-cert\") pod \"authorino-5f9995dd6f-pppw8\" (UID: \"a563ddad-4fab-4b9f-9e01-c325efed519a\") " pod="kuadrant-system/authorino-5f9995dd6f-pppw8"
Nov 01 12:33:41.827636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.827607    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctqc\" (UniqueName: \"kubernetes.io/projected/a563ddad-4fab-4b9f-9e01-c325efed519a-kube-api-access-sctqc\") pod \"authorino-5f9995dd6f-pppw8\" (UID: \"a563ddad-4fab-4b9f-9e01-c325efed519a\") " pod="kuadrant-system/authorino-5f9995dd6f-pppw8"
Nov 01 12:33:41.915634 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:41.915552    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9995dd6f-pppw8"
Nov 01 12:33:42.284633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:42.284603    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f9995dd6f-pppw8"]
Nov 01 12:33:42.292607 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:33:42.292572    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda563ddad_4fab_4b9f_9e01_c325efed519a.slice/crio-83e31a161ba0ca7baca1907fd14d565fade5588fe986ea0336bd429e05e146fd WatchSource:0}: Error finding container 83e31a161ba0ca7baca1907fd14d565fade5588fe986ea0336bd429e05e146fd: Status 404 returned error can't find the container with id 83e31a161ba0ca7baca1907fd14d565fade5588fe986ea0336bd429e05e146fd
Nov 01 12:33:42.293890 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:42.293872    5725 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:33:42.888196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:42.888157    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9995dd6f-pppw8" event={"ID":"a563ddad-4fab-4b9f-9e01-c325efed519a","Type":"ContainerStarted","Data":"83e31a161ba0ca7baca1907fd14d565fade5588fe986ea0336bd429e05e146fd"}
Nov 01 12:33:44.902865 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:44.902819    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9995dd6f-pppw8" event={"ID":"a563ddad-4fab-4b9f-9e01-c325efed519a","Type":"ContainerStarted","Data":"19cee9d101c1ab6a17786ace141d25a36758f0cd7ac1006f7994bde7db4d8ad6"}
Nov 01 12:33:45.030773 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.030701    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f9995dd6f-pppw8" podStartSLOduration=2.403717674 podStartE2EDuration="4.030680446s" totalImagesPullingTime="1.626962772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:33:45.028995879 +0000 UTC m=+666.536187097" watchObservedRunningTime="2026-11-01 12:33:45.030680446 +0000 UTC m=+666.537871666"
Nov 01 12:33:45.081130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.081093    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f45579c86-g6wk7"]
Nov 01 12:33:45.082541 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.082482    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f45579c86-g6wk7" podUID="89618e46-2fde-41aa-9eea-6b4fb6fe2413" containerName="authorino" containerID="cri-o://5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d" gracePeriod=30
Nov 01 12:33:45.379213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.379182    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:33:45.422325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.422278    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89618e46-2fde-41aa-9eea-6b4fb6fe2413-tls-cert\" (UniqueName: \"kubernetes.io/secret/89618e46-2fde-41aa-9eea-6b4fb6fe2413-tls-cert\") pod \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\" (UID: \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\") "
Nov 01 12:33:45.422503 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.422401    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89618e46-2fde-41aa-9eea-6b4fb6fe2413-kube-api-access-cf6lh\" (UniqueName: \"kubernetes.io/projected/89618e46-2fde-41aa-9eea-6b4fb6fe2413-kube-api-access-cf6lh\") pod \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\" (UID: \"89618e46-2fde-41aa-9eea-6b4fb6fe2413\") "
Nov 01 12:33:45.424431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.424404    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89618e46-2fde-41aa-9eea-6b4fb6fe2413-kube-api-access-cf6lh" pod "89618e46-2fde-41aa-9eea-6b4fb6fe2413" (UID: "89618e46-2fde-41aa-9eea-6b4fb6fe2413"). InnerVolumeSpecName "kube-api-access-cf6lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:33:45.433950 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.433919    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89618e46-2fde-41aa-9eea-6b4fb6fe2413-tls-cert" pod "89618e46-2fde-41aa-9eea-6b4fb6fe2413" (UID: "89618e46-2fde-41aa-9eea-6b4fb6fe2413"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:33:45.523869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.523831    5725 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89618e46-2fde-41aa-9eea-6b4fb6fe2413-tls-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:33:45.523869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.523863    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6lh\" (UniqueName: \"kubernetes.io/projected/89618e46-2fde-41aa-9eea-6b4fb6fe2413-kube-api-access-cf6lh\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:33:45.909794 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.909696    5725 generic.go:356] "Generic (PLEG): container finished" podID="89618e46-2fde-41aa-9eea-6b4fb6fe2413" containerID="5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d" exitCode=0
Nov 01 12:33:45.909794 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.909737    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f45579c86-g6wk7"
Nov 01 12:33:45.910346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.909785    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f45579c86-g6wk7" event={"ID":"89618e46-2fde-41aa-9eea-6b4fb6fe2413","Type":"ContainerDied","Data":"5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d"}
Nov 01 12:33:45.910346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.909834    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f45579c86-g6wk7" event={"ID":"89618e46-2fde-41aa-9eea-6b4fb6fe2413","Type":"ContainerDied","Data":"7f63854148cf0e428cf753877dfeb265e521de1c728ad1feac2354d26a712702"}
Nov 01 12:33:45.910346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.909856    5725 scope.go:122] "RemoveContainer" containerID="5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d"
Nov 01 12:33:45.920249 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.920182    5725 scope.go:122] "RemoveContainer" containerID="5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d"
Nov 01 12:33:45.920513 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:33:45.920493    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d\": container with ID starting with 5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d not found: ID does not exist" containerID="5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d"
Nov 01 12:33:45.920581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:45.920527    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d"} err="failed to get container status \"5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d\": rpc error: code = NotFound desc = could not find container \"5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d\": container with ID starting with 5c56bf13550559190487fbf5b1f2dd0d7660d8fcf83ab729980d14d97e192d9d not found: ID does not exist"
Nov 01 12:33:46.069032 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:46.068973    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f45579c86-g6wk7"]
Nov 01 12:33:46.071194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:46.071163    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f45579c86-g6wk7"]
Nov 01 12:33:46.616788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:33:46.616752    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89618e46-2fde-41aa-9eea-6b4fb6fe2413" path="/var/lib/kubelet/pods/89618e46-2fde-41aa-9eea-6b4fb6fe2413/volumes"
Nov 01 12:34:01.029105 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:34:01.029045    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:34:31.026685 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:34:31.026637    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:35:01.028469 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:35:01.028386    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:35:03.616066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.616024    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"]
Nov 01 12:35:03.616487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.616472    5725 cpu_manager.go:414] "containerMap: removing container" podUID="89618e46-2fde-41aa-9eea-6b4fb6fe2413" containerName="authorino"
Nov 01 12:35:03.616487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.616488    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89618e46-2fde-41aa-9eea-6b4fb6fe2413" containerName="authorino"
Nov 01 12:35:03.616602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.616592    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="89618e46-2fde-41aa-9eea-6b4fb6fe2413" containerName="authorino"
Nov 01 12:35:03.619917 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.619890    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:03.672534 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.672497    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"]
Nov 01 12:35:03.728492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.728418    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-06fd6b\""
Nov 01 12:35:03.728916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.728893    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-06fd6b-dockercfg-pzj2g\""
Nov 01 12:35:03.731014 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.730975    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc6f9cc-7039-4174-967c-d264de6c076e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv\" (UID: \"2cc6f9cc-7039-4174-967c-d264de6c076e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:03.731400 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.731378    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtr4q\" (UniqueName: \"kubernetes.io/projected/2cc6f9cc-7039-4174-967c-d264de6c076e-kube-api-access-gtr4q\") pod \"payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv\" (UID: \"2cc6f9cc-7039-4174-967c-d264de6c076e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:03.833027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.832979    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtr4q\" (UniqueName: \"kubernetes.io/projected/2cc6f9cc-7039-4174-967c-d264de6c076e-kube-api-access-gtr4q\") pod \"payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv\" (UID: \"2cc6f9cc-7039-4174-967c-d264de6c076e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:03.833251 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.833200    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc6f9cc-7039-4174-967c-d264de6c076e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv\" (UID: \"2cc6f9cc-7039-4174-967c-d264de6c076e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:03.834800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.834770    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc6f9cc-7039-4174-967c-d264de6c076e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv\" (UID: \"2cc6f9cc-7039-4174-967c-d264de6c076e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:03.888009 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.887934    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtr4q\" (UniqueName: \"kubernetes.io/projected/2cc6f9cc-7039-4174-967c-d264de6c076e-kube-api-access-gtr4q\") pod \"payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv\" (UID: \"2cc6f9cc-7039-4174-967c-d264de6c076e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:03.933915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:03.933870    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:04.310294 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:04.310253    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"]
Nov 01 12:35:04.394948 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:04.394912    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv" event={"ID":"2cc6f9cc-7039-4174-967c-d264de6c076e","Type":"ContainerStarted","Data":"b4f278f4b971f2a5ae5072a6b51528304eb961ae2d0bc70cc3e20653510d9f88"}
Nov 01 12:35:06.101699 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.101661    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"]
Nov 01 12:35:06.105494 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.105466    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:06.159801 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.158857    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"]
Nov 01 12:35:06.159801 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.159381    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2n6\" (UniqueName: \"kubernetes.io/projected/87de366f-a30e-43f0-90ec-9671c04de355-kube-api-access-nc2n6\") pod \"payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn\" (UID: \"87de366f-a30e-43f0-90ec-9671c04de355\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:06.159801 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.159468    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de366f-a30e-43f0-90ec-9671c04de355-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn\" (UID: \"87de366f-a30e-43f0-90ec-9671c04de355\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:06.210109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.202315    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-f7427d-dockercfg-nw6pg\""
Nov 01 12:35:06.223072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.221730    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-f7427d\""
Nov 01 12:35:06.257757 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.257719    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"]
Nov 01 12:35:06.260811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.260771    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de366f-a30e-43f0-90ec-9671c04de355-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn\" (UID: \"87de366f-a30e-43f0-90ec-9671c04de355\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:06.261039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.260878    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2n6\" (UniqueName: \"kubernetes.io/projected/87de366f-a30e-43f0-90ec-9671c04de355-kube-api-access-nc2n6\") pod \"payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn\" (UID: \"87de366f-a30e-43f0-90ec-9671c04de355\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:06.261614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.261589    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de366f-a30e-43f0-90ec-9671c04de355-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn\" (UID: \"87de366f-a30e-43f0-90ec-9671c04de355\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:06.261703 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.261675    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:06.314959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.314918    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"]
Nov 01 12:35:06.331279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.331243    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2n6\" (UniqueName: \"kubernetes.io/projected/87de366f-a30e-43f0-90ec-9671c04de355-kube-api-access-nc2n6\") pod \"payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn\" (UID: \"87de366f-a30e-43f0-90ec-9671c04de355\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:06.362484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.362404    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e490d69c-9098-4829-9e8d-14e448dc5f74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr\" (UID: \"e490d69c-9098-4829-9e8d-14e448dc5f74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:06.362484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.362447    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mgk8\" (UniqueName: \"kubernetes.io/projected/e490d69c-9098-4829-9e8d-14e448dc5f74-kube-api-access-8mgk8\") pod \"payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr\" (UID: \"e490d69c-9098-4829-9e8d-14e448dc5f74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:06.423095 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.423044    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:06.463266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.463219    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e490d69c-9098-4829-9e8d-14e448dc5f74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr\" (UID: \"e490d69c-9098-4829-9e8d-14e448dc5f74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:06.463453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.463282    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mgk8\" (UniqueName: \"kubernetes.io/projected/e490d69c-9098-4829-9e8d-14e448dc5f74-kube-api-access-8mgk8\") pod \"payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr\" (UID: \"e490d69c-9098-4829-9e8d-14e448dc5f74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:06.464292 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.464267    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e490d69c-9098-4829-9e8d-14e448dc5f74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr\" (UID: \"e490d69c-9098-4829-9e8d-14e448dc5f74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:06.518786 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.518751    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mgk8\" (UniqueName: \"kubernetes.io/projected/e490d69c-9098-4829-9e8d-14e448dc5f74-kube-api-access-8mgk8\") pod \"payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr\" (UID: \"e490d69c-9098-4829-9e8d-14e448dc5f74\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:06.576578 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.576534    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:06.815005 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.814966    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"]
Nov 01 12:35:06.821798 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:35:06.821764    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87de366f_a30e_43f0_90ec_9671c04de355.slice/crio-c35f9c89f9fed2fc30dfe5f8e061f6959b10f2a4fdfcdc8cd1e5ae92474a338f WatchSource:0}: Error finding container c35f9c89f9fed2fc30dfe5f8e061f6959b10f2a4fdfcdc8cd1e5ae92474a338f: Status 404 returned error can't find the container with id c35f9c89f9fed2fc30dfe5f8e061f6959b10f2a4fdfcdc8cd1e5ae92474a338f
Nov 01 12:35:06.964368 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:06.964283    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"]
Nov 01 12:35:06.972025 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:35:06.971985    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode490d69c_9098_4829_9e8d_14e448dc5f74.slice/crio-8320dfa9bbaaf59cc58a61776dcf30c3c0307a7e653297412f1cd3c29a718dba WatchSource:0}: Error finding container 8320dfa9bbaaf59cc58a61776dcf30c3c0307a7e653297412f1cd3c29a718dba: Status 404 returned error can't find the container with id 8320dfa9bbaaf59cc58a61776dcf30c3c0307a7e653297412f1cd3c29a718dba
Nov 01 12:35:07.416983 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:07.416941    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn" event={"ID":"87de366f-a30e-43f0-90ec-9671c04de355","Type":"ContainerStarted","Data":"c35f9c89f9fed2fc30dfe5f8e061f6959b10f2a4fdfcdc8cd1e5ae92474a338f"}
Nov 01 12:35:07.418659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:07.418629    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv" event={"ID":"2cc6f9cc-7039-4174-967c-d264de6c076e","Type":"ContainerStarted","Data":"da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43"}
Nov 01 12:35:07.418795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:07.418721    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:07.419925 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:07.419902    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr" event={"ID":"e490d69c-9098-4829-9e8d-14e448dc5f74","Type":"ContainerStarted","Data":"8320dfa9bbaaf59cc58a61776dcf30c3c0307a7e653297412f1cd3c29a718dba"}
Nov 01 12:35:07.526856 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:07.526741    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv" podStartSLOduration=1.576634042 podStartE2EDuration="4.526726771s" totalImagesPullingTime="2.950092729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:07.524858237 +0000 UTC m=+749.032049454" watchObservedRunningTime="2026-11-01 12:35:07.526726771 +0000 UTC m=+749.033918022"
Nov 01 12:35:08.427331 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:08.427280    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr" event={"ID":"e490d69c-9098-4829-9e8d-14e448dc5f74","Type":"ContainerStarted","Data":"7c5c21beb02f31e80ee88104b507ead9b1d4370246a4725f116c1505bad86cf1"}
Nov 01 12:35:08.427812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:08.427446    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:08.428822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:08.428800    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn" event={"ID":"87de366f-a30e-43f0-90ec-9671c04de355","Type":"ContainerStarted","Data":"2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117"}
Nov 01 12:35:08.428934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:08.428853    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:08.534112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:08.534046    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr" podStartSLOduration=1.66174782 podStartE2EDuration="2.53403099s" totalImagesPullingTime="872.28317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:08.532999137 +0000 UTC m=+750.040190353" watchObservedRunningTime="2026-11-01 12:35:08.53403099 +0000 UTC m=+750.041222238"
Nov 01 12:35:08.643140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:08.643032    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn" podStartSLOduration=1.840704898 podStartE2EDuration="2.64301447s" totalImagesPullingTime="802.309572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:08.639467648 +0000 UTC m=+750.146658866" watchObservedRunningTime="2026-11-01 12:35:08.64301447 +0000 UTC m=+750.150205688"
Nov 01 12:35:10.911007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:10.910969    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"]
Nov 01 12:35:10.914930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:10.914912    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:10.969214 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:10.969175    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"]
Nov 01 12:35:11.020353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.020315    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-9514b0-dockercfg-bf58v\""
Nov 01 12:35:11.020525 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.020416    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-9514b0\""
Nov 01 12:35:11.021595 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.021571    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c585c6-9d66-4a3c-b13e-c6004c079867-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5\" (UID: \"a0c585c6-9d66-4a3c-b13e-c6004c079867\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:11.021710 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.021626    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sh49\" (UniqueName: \"kubernetes.io/projected/a0c585c6-9d66-4a3c-b13e-c6004c079867-kube-api-access-6sh49\") pod \"payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5\" (UID: \"a0c585c6-9d66-4a3c-b13e-c6004c079867\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:11.059676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.059643    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"]
Nov 01 12:35:11.063553 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.063534    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:11.117004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.116961    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"]
Nov 01 12:35:11.122809 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.122767    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cb0bff-44e1-409a-b7fe-891015b30fcb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6\" (UID: \"40cb0bff-44e1-409a-b7fe-891015b30fcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:11.123004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.122937    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c585c6-9d66-4a3c-b13e-c6004c079867-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5\" (UID: \"a0c585c6-9d66-4a3c-b13e-c6004c079867\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:11.123004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.122985    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcw9j\" (UniqueName: \"kubernetes.io/projected/40cb0bff-44e1-409a-b7fe-891015b30fcb-kube-api-access-wcw9j\") pod \"payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6\" (UID: \"40cb0bff-44e1-409a-b7fe-891015b30fcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:11.123187 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.123027    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sh49\" (UniqueName: \"kubernetes.io/projected/a0c585c6-9d66-4a3c-b13e-c6004c079867-kube-api-access-6sh49\") pod \"payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5\" (UID: \"a0c585c6-9d66-4a3c-b13e-c6004c079867\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:11.123735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.123711    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c585c6-9d66-4a3c-b13e-c6004c079867-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5\" (UID: \"a0c585c6-9d66-4a3c-b13e-c6004c079867\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:11.177349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.177271    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sh49\" (UniqueName: \"kubernetes.io/projected/a0c585c6-9d66-4a3c-b13e-c6004c079867-kube-api-access-6sh49\") pod \"payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5\" (UID: \"a0c585c6-9d66-4a3c-b13e-c6004c079867\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:11.224500 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.224463    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cb0bff-44e1-409a-b7fe-891015b30fcb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6\" (UID: \"40cb0bff-44e1-409a-b7fe-891015b30fcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:11.224688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.224566    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcw9j\" (UniqueName: \"kubernetes.io/projected/40cb0bff-44e1-409a-b7fe-891015b30fcb-kube-api-access-wcw9j\") pod \"payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6\" (UID: \"40cb0bff-44e1-409a-b7fe-891015b30fcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:11.225242 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.225223    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cb0bff-44e1-409a-b7fe-891015b30fcb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6\" (UID: \"40cb0bff-44e1-409a-b7fe-891015b30fcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:11.228363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.228339    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:11.278835 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.278433    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcw9j\" (UniqueName: \"kubernetes.io/projected/40cb0bff-44e1-409a-b7fe-891015b30fcb-kube-api-access-wcw9j\") pod \"payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6\" (UID: \"40cb0bff-44e1-409a-b7fe-891015b30fcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:11.379954 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.379913    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:11.614133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.614095    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"]
Nov 01 12:35:11.757750 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:11.757712    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"]
Nov 01 12:35:11.763668 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:35:11.763634    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cb0bff_44e1_409a_b7fe_891015b30fcb.slice/crio-a24b8b024066bcfdc1a26137cfe00c0a36cc20acd9ce83911d02ea96399e54c4 WatchSource:0}: Error finding container a24b8b024066bcfdc1a26137cfe00c0a36cc20acd9ce83911d02ea96399e54c4: Status 404 returned error can't find the container with id a24b8b024066bcfdc1a26137cfe00c0a36cc20acd9ce83911d02ea96399e54c4
Nov 01 12:35:12.462372 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:12.462334    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5" event={"ID":"a0c585c6-9d66-4a3c-b13e-c6004c079867","Type":"ContainerStarted","Data":"9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4"}
Nov 01 12:35:12.462867 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:12.462401    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5" event={"ID":"a0c585c6-9d66-4a3c-b13e-c6004c079867","Type":"ContainerStarted","Data":"2696e552b40d792a5671e038c703e8680c414bfd3fd5d00d173558425d3b8ba1"}
Nov 01 12:35:12.462867 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:12.462549    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:12.464272 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:12.464249    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6" event={"ID":"40cb0bff-44e1-409a-b7fe-891015b30fcb","Type":"ContainerStarted","Data":"729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e"}
Nov 01 12:35:12.464366 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:12.464290    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6" event={"ID":"40cb0bff-44e1-409a-b7fe-891015b30fcb","Type":"ContainerStarted","Data":"a24b8b024066bcfdc1a26137cfe00c0a36cc20acd9ce83911d02ea96399e54c4"}
Nov 01 12:35:12.464366 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:12.464326    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:12.569459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:12.569397    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5" podStartSLOduration=2.569374942 podStartE2EDuration="2.569374942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:12.568468557 +0000 UTC m=+754.075659774" watchObservedRunningTime="2026-11-01 12:35:12.569374942 +0000 UTC m=+754.076566189"
Nov 01 12:35:12.676129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:12.676066    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6" podStartSLOduration=1.676040156 podStartE2EDuration="1.676040156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:12.674445102 +0000 UTC m=+754.181636319" watchObservedRunningTime="2026-11-01 12:35:12.676040156 +0000 UTC m=+754.183231372"
Nov 01 12:35:13.241435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.241399    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"]
Nov 01 12:35:13.245957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.245931    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.343868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.343396    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c7206f-openshift-default-dockercfg-8pj7g\""
Nov 01 12:35:13.345732 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.345702    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.345866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.345773    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.345940 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.345910    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.346013 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.345949    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.346013 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.345993    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.346149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.346027    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vr4\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-kube-api-access-c4vr4\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.346149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.346078    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.346149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.346106    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.346350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.346157    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.348034 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.348010    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"]
Nov 01 12:35:13.447698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.447664    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.447918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.447726    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.447918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.447774    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.447918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.447799    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.447918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.447835    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.447918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.447864    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vr4\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-kube-api-access-c4vr4\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.447918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.447901    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.448294 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.447926    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.448294 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.447950    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.448294 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.448166    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.448451 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.448346    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.448514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.448497    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.448642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.448624    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.448766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.448736    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.450762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.450733    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.450910 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.450892    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.504351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.504280    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.504351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.504291    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vr4\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-kube-api-access-c4vr4\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:13.577267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:13.566252    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:14.012007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:14.011970    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"]
Nov 01 12:35:14.019041 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:35:14.019009    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaaaf7c_b849_44e5_8ea3_d042a0bbed18.slice/crio-d7330fa6ef749d2f6d48ede2cc9b69104e50d4801b396186d16e54d5f7fb34d2 WatchSource:0}: Error finding container d7330fa6ef749d2f6d48ede2cc9b69104e50d4801b396186d16e54d5f7fb34d2: Status 404 returned error can't find the container with id d7330fa6ef749d2f6d48ede2cc9b69104e50d4801b396186d16e54d5f7fb34d2
Nov 01 12:35:14.021515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:14.021481    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:35:14.021594 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:14.021555    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:35:14.021594 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:14.021586    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:35:14.480748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:14.480713    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" event={"ID":"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18","Type":"ContainerStarted","Data":"1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a"}
Nov 01 12:35:14.480951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:14.480765    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" event={"ID":"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18","Type":"ContainerStarted","Data":"d7330fa6ef749d2f6d48ede2cc9b69104e50d4801b396186d16e54d5f7fb34d2"}
Nov 01 12:35:14.568369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:14.568337    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:14.593588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:14.593528    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" podStartSLOduration=1.5935122640000001 podStartE2EDuration="1.593512264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:14.591615405 +0000 UTC m=+756.098806650" watchObservedRunningTime="2026-11-01 12:35:14.593512264 +0000 UTC m=+756.100703482"
Nov 01 12:35:15.570088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:15.570037    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 12:35:15.570592 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:15.570364    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 12:35:16.568842 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:16.568801    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 12:35:16.569081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:16.568865    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 12:35:17.568778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:17.568731    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 12:35:17.569323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:17.568790    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 12:35:18.448269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:18.448235    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 12:35:18.568651 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:18.568614    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 12:35:18.568853 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:18.568675    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 12:35:19.456077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:19.455395    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 12:35:19.458554 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:19.458529    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 12:35:19.568633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:19.568592    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 12:35:19.568852 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:19.568661    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 12:35:20.573432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:20.573385    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:20.573865 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:20.573802    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:20.574715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:20.574692    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:35:23.474665 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:23.474632    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 12:35:23.475166 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:23.474789    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 12:35:28.701178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.701143    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"]
Nov 01 12:35:28.705238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.705215    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.729102 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.729042    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.729288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.729127    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.729288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.729157    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.729288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.729258    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.729471 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.729294    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417ed549-3aaa-47e6-b135-fa4317813918-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.729471 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.729312    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.729471 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.729332    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.729471 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.729347    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417ed549-3aaa-47e6-b135-fa4317813918-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.729471 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.729445    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdsx8\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-kube-api-access-gdsx8\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.749368 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.749335    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"]
Nov 01 12:35:28.753790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.753767    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.763574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.763545    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-06fd6b-openshift-default-dockercfg-6x29r\""
Nov 01 12:35:28.795041 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.795006    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"]
Nov 01 12:35:28.807749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.807715    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"]
Nov 01 12:35:28.830253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830199    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdsx8\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-kube-api-access-gdsx8\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.830253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830278    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkcd\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-kube-api-access-ngkcd\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830311    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d541627-125c-4a48-b599-129c78b75c94-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830353    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830378    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830403    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830430    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d541627-125c-4a48-b599-129c78b75c94-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830484    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830526    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830550    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830582    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.830614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830615    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.831444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830654    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417ed549-3aaa-47e6-b135-fa4317813918-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.831444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830682    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.831444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830721    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.831444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830747    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417ed549-3aaa-47e6-b135-fa4317813918-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.831444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830777    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.831444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.830826    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.831838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.831598    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.831918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.831858    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.832749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.832724    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.833242 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.833198    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.833602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.833575    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417ed549-3aaa-47e6-b135-fa4317813918-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.834798 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.834771    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.838318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.838292    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417ed549-3aaa-47e6-b135-fa4317813918-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.889026 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.888986    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdsx8\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-kube-api-access-gdsx8\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.889221 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.889069    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:28.931896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.931857    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932120 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.931933    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932120 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.931969    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932120 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932029    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkcd\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-kube-api-access-ngkcd\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932120 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932089    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d541627-125c-4a48-b599-129c78b75c94-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932134    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d541627-125c-4a48-b599-129c78b75c94-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932185    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932228    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932256    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932341    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932439    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.932925 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932869    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.933062 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.932966    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.933062 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.933020    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d541627-125c-4a48-b599-129c78b75c94-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.934642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.934622    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.935366 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.935349    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d541627-125c-4a48-b599-129c78b75c94-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.991254 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.991224    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:28.991448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:28.991382    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkcd\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-kube-api-access-ngkcd\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:29.023279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.023043    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:29.071372 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.071335    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:29.462008 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.461919    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"]
Nov 01 12:35:29.468398 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:35:29.468363    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417ed549_3aaa_47e6_b135_fa4317813918.slice/crio-9ecd4702c40619dffa0aec3ba8d6d1c503300e25438c5f08cb97445d2e47c3d1 WatchSource:0}: Error finding container 9ecd4702c40619dffa0aec3ba8d6d1c503300e25438c5f08cb97445d2e47c3d1: Status 404 returned error can't find the container with id 9ecd4702c40619dffa0aec3ba8d6d1c503300e25438c5f08cb97445d2e47c3d1
Nov 01 12:35:29.470870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.470832    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:35:29.470978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.470920    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:35:29.470978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.470963    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:35:29.520976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.520941    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"]
Nov 01 12:35:29.528089 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:35:29.528019    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d541627_125c_4a48_b599_129c78b75c94.slice/crio-e4fc4629f2c76595fb88d64dbdbcaf8cf5e557ba665fbea4822f6d68f241045b WatchSource:0}: Error finding container e4fc4629f2c76595fb88d64dbdbcaf8cf5e557ba665fbea4822f6d68f241045b: Status 404 returned error can't find the container with id e4fc4629f2c76595fb88d64dbdbcaf8cf5e557ba665fbea4822f6d68f241045b
Nov 01 12:35:29.539293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.535074    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:35:29.539293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.535238    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:35:29.539293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.535334    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:35:29.586064 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.586017    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" event={"ID":"417ed549-3aaa-47e6-b135-fa4317813918","Type":"ContainerStarted","Data":"85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b"}
Nov 01 12:35:29.586293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.586271    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" event={"ID":"417ed549-3aaa-47e6-b135-fa4317813918","Type":"ContainerStarted","Data":"9ecd4702c40619dffa0aec3ba8d6d1c503300e25438c5f08cb97445d2e47c3d1"}
Nov 01 12:35:29.587644 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.587621    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" event={"ID":"7d541627-125c-4a48-b599-129c78b75c94","Type":"ContainerStarted","Data":"e4fc4629f2c76595fb88d64dbdbcaf8cf5e557ba665fbea4822f6d68f241045b"}
Nov 01 12:35:29.696186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:29.696125    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" podStartSLOduration=1.696110851 podStartE2EDuration="1.696110851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:29.695114867 +0000 UTC m=+771.202306084" watchObservedRunningTime="2026-11-01 12:35:29.696110851 +0000 UTC m=+771.203302068"
Nov 01 12:35:30.024924 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:30.024058    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:30.025501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:30.025471    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 12:35:30.025581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:30.025531    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 12:35:30.598945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:30.598901    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" event={"ID":"7d541627-125c-4a48-b599-129c78b75c94","Type":"ContainerStarted","Data":"1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56"}
Nov 01 12:35:30.710906 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:30.710829    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" podStartSLOduration=2.710814163 podStartE2EDuration="2.710814163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:30.708361002 +0000 UTC m=+772.215552218" watchObservedRunningTime="2026-11-01 12:35:30.710814163 +0000 UTC m=+772.218005380"
Nov 01 12:35:31.024070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:31.023770    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 12:35:31.024070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:31.023831    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 12:35:31.026731 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:35:31.026704    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:35:31.083074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:31.075754    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:31.083074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:31.075901    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:35:31.083074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:31.075942    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 12:35:32.024425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:32.024031    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 12:35:32.024425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:32.024105    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 12:35:32.072542 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:32.072507    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:35:32.073139 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:32.073110    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 12:35:33.023965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:33.023929    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 12:35:33.024182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:33.023994    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 12:35:33.077076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:33.076472    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:35:33.077076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:33.076537    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 12:35:34.024163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:34.024131    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 12:35:34.024351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:34.024178    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 12:35:34.076086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:34.075460    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:35:34.076086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:34.075525    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 12:35:35.024313 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:35.024271    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 12:35:35.024792 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:35.024331    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 12:35:35.078076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:35.076921    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:35:35.078076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:35.076978    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 12:35:36.028905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:36.028872    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:36.029324 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:36.029167    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:36.029994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:36.029970    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:35:36.076190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:36.076163    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:36.076569 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:36.076549    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:36.077257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:36.077241    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:35:59.974364 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:59.974316    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"]
Nov 01 12:35:59.978077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:35:59.978039    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.030109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.030035    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f7427d-openshift-default-dockercfg-zdm4c\""
Nov 01 12:36:00.062884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.062847    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.062884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.062888    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.063196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.062960    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.063196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.062997    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.063196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.063017    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.063196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.063035    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.063196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.063074    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.063196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.063107    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-kube-api-access-j5ghv\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.063196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.063126    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.077999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.077958    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"]
Nov 01 12:36:00.163884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.163839    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-kube-api-access-j5ghv\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.163900    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.163954    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.163972    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.164092    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.164145    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.164171    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.164202    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.164247    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164551 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.164495    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164941 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.164701    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164941 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.164799    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.164941 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.164896    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.165233 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.165123    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.166794 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.166773    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.167216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.167189    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.219280 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.219248    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.219587 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.219544    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-kube-api-access-j5ghv\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.292328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.292240    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:00.743524 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.743486    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"]
Nov 01 12:36:00.750149 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:36:00.750121    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc384213_bc6e_4d61_943a_5e88e4eb1ea4.slice/crio-1c383a28d3ba490eee66586f778f438f893ea2fa743e4664b7012c0083719d95 WatchSource:0}: Error finding container 1c383a28d3ba490eee66586f778f438f893ea2fa743e4664b7012c0083719d95: Status 404 returned error can't find the container with id 1c383a28d3ba490eee66586f778f438f893ea2fa743e4664b7012c0083719d95
Nov 01 12:36:00.752496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.752463    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:36:00.752575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.752528    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:36:00.752575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.752561    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:36:00.794523 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:00.794351    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" event={"ID":"dc384213-bc6e-4d61-943a-5e88e4eb1ea4","Type":"ContainerStarted","Data":"1c383a28d3ba490eee66586f778f438f893ea2fa743e4664b7012c0083719d95"}
Nov 01 12:36:01.027809 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:36:01.027696    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:36:01.803210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:01.803174    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" event={"ID":"dc384213-bc6e-4d61-943a-5e88e4eb1ea4","Type":"ContainerStarted","Data":"f13cfc0ba614cc28466330be96ab29221b11f8d32a0e9173f0ed932de8831e72"}
Nov 01 12:36:01.944128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:01.942795    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" podStartSLOduration=2.94276933 podStartE2EDuration="2.94276933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:01.939659345 +0000 UTC m=+803.446850605" watchObservedRunningTime="2026-11-01 12:36:01.94276933 +0000 UTC m=+803.449960549"
Nov 01 12:36:02.293434 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:02.293397    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:02.295203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:02.295176    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 12:36:02.295347 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:02.295234    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 12:36:03.293438 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:03.293402    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 12:36:03.293865 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:03.293472    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 12:36:04.293566 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:04.293517    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 12:36:04.294129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:04.293583    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 12:36:05.293283 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:05.293250    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 12:36:05.293453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:05.293301    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 12:36:06.292798 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:06.292760    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 12:36:06.293208 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:06.292840    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 12:36:07.297280 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:07.297252    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:07.297699 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:07.297528    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:07.298343 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:07.298319    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 12:36:24.299722 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.299629    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"]
Nov 01 12:36:24.303672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.303648    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.357276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.357240    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"]
Nov 01 12:36:24.383424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.383396    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-home\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.383576 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.383533    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-dshm\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.383642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.383588    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tls-certs\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.383642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.383623    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndlr\" (UniqueName: \"kubernetes.io/projected/cd8dec32-afbe-4385-a7e9-d57fc37848ac-kube-api-access-mndlr\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.383748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.383709    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tmp-dir\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.383808 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.383792    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-model-cache\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.400343 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.400310    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"openshift-service-ca.crt\""
Nov 01 12:36:24.400496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.400357    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"kube-root-ca.crt\""
Nov 01 12:36:24.400496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.400437    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"simulator-w4-06fd6b-kserve-self-signed-certs\""
Nov 01 12:36:24.400496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.400491    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"default-dockercfg-l9jz2\""
Nov 01 12:36:24.485409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.485375    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-dshm\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.485605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.485419    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tls-certs\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.485605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.485440    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndlr\" (UniqueName: \"kubernetes.io/projected/cd8dec32-afbe-4385-a7e9-d57fc37848ac-kube-api-access-mndlr\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.485605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.485469    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tmp-dir\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.485605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.485517    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-model-cache\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.485605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.485561    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-home\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.486040 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.486010    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-model-cache\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.486182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.486100    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tmp-dir\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.486182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.486108    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-home\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.487884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.487860    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-dshm\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.488841 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.488822    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tls-certs\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.539921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.539885    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndlr\" (UniqueName: \"kubernetes.io/projected/cd8dec32-afbe-4385-a7e9-d57fc37848ac-kube-api-access-mndlr\") pod \"simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") " pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:24.640631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:24.640529    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:25.021087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:25.021015    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"]
Nov 01 12:36:25.028137 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:36:25.028107    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd8dec32_afbe_4385_a7e9_d57fc37848ac.slice/crio-63570572789b83e41ed257f71ad0431961d384a0bfa6a042cc8f73c524ee48ec WatchSource:0}: Error finding container 63570572789b83e41ed257f71ad0431961d384a0bfa6a042cc8f73c524ee48ec: Status 404 returned error can't find the container with id 63570572789b83e41ed257f71ad0431961d384a0bfa6a042cc8f73c524ee48ec
Nov 01 12:36:25.990305 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:25.990267    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" event={"ID":"cd8dec32-afbe-4385-a7e9-d57fc37848ac","Type":"ContainerStarted","Data":"63570572789b83e41ed257f71ad0431961d384a0bfa6a042cc8f73c524ee48ec"}
Nov 01 12:36:28.006925 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:28.006888    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" event={"ID":"cd8dec32-afbe-4385-a7e9-d57fc37848ac","Type":"ContainerStarted","Data":"56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057"}
Nov 01 12:36:28.115483 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:28.115431    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podStartSLOduration=1.747060001 podStartE2EDuration="4.115416648s" totalImagesPullingTime="2.368356647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:28.114336226 +0000 UTC m=+829.621527563" watchObservedRunningTime="2026-11-01 12:36:28.115416648 +0000 UTC m=+829.622607864"
Nov 01 12:36:28.641135 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:28.641102    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:31.034930 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:36:31.034891    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:36:34.641538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:34.641501    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:34.655515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:34.655486    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:35.071012 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:35.070981    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:36:52.920945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:52.920910    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"]
Nov 01 12:36:52.926158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:52.926135    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:52.978574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:52.978534    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"]
Nov 01 12:36:53.012563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.012526    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-model-cache\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.012563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.012565    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-home\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.012769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.012688    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85276343-99cd-427a-b0b2-2eea7e0951bf-tls-certs\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.012769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.012746    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs2nz\" (UniqueName: \"kubernetes.io/projected/85276343-99cd-427a-b0b2-2eea7e0951bf-kube-api-access-cs2nz\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.012867 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.012846    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-tmp-dir\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.012914 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.012895    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-dshm\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.021700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.021659    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"kube-root-ca.crt\""
Nov 01 12:36:53.021700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.021683    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"openshift-service-ca.crt\""
Nov 01 12:36:53.022827 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.022809    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"default-dockercfg-rrgs4\""
Nov 01 12:36:53.022908 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.022841    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"simulator-w5-9514b0-kserve-self-signed-certs\""
Nov 01 12:36:53.113837 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.113801    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-model-cache\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.114070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.113844    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-home\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.114070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.113952    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85276343-99cd-427a-b0b2-2eea7e0951bf-tls-certs\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.114070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.114032    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs2nz\" (UniqueName: \"kubernetes.io/projected/85276343-99cd-427a-b0b2-2eea7e0951bf-kube-api-access-cs2nz\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.114258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.114114    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-tmp-dir\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.114258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.114151    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-dshm\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.114362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.114339    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-home\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.114362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.114354    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-model-cache\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.114898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.114866    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-tmp-dir\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.117545 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.116960    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85276343-99cd-427a-b0b2-2eea7e0951bf-tls-certs\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.118153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.118125    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-dshm\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.168247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.168214    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs2nz\" (UniqueName: \"kubernetes.io/projected/85276343-99cd-427a-b0b2-2eea7e0951bf-kube-api-access-cs2nz\") pod \"simulator-w5-9514b0-kserve-65d6557df5-ml6d4\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") " pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.238908 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.238856    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:53.633712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:53.633632    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"]
Nov 01 12:36:53.640936 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:36:53.640906    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85276343_99cd_427a_b0b2_2eea7e0951bf.slice/crio-cacecdb09033ad771df761e82c4cf4369abca3cfcefe0672960515c216980c33 WatchSource:0}: Error finding container cacecdb09033ad771df761e82c4cf4369abca3cfcefe0672960515c216980c33: Status 404 returned error can't find the container with id cacecdb09033ad771df761e82c4cf4369abca3cfcefe0672960515c216980c33
Nov 01 12:36:54.187563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:54.187525    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" event={"ID":"85276343-99cd-427a-b0b2-2eea7e0951bf","Type":"ContainerStarted","Data":"1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6"}
Nov 01 12:36:54.187951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:54.187564    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" event={"ID":"85276343-99cd-427a-b0b2-2eea7e0951bf","Type":"ContainerStarted","Data":"cacecdb09033ad771df761e82c4cf4369abca3cfcefe0672960515c216980c33"}
Nov 01 12:36:54.239568 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:54.239535    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:36:54.295471 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:36:54.295417    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podStartSLOduration=2.295402964 podStartE2EDuration="2.295402964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:54.29484456 +0000 UTC m=+855.802035777" watchObservedRunningTime="2026-11-01 12:36:54.295402964 +0000 UTC m=+855.802594181"
Nov 01 12:37:01.109341 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:37:01.109243    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:37:03.239475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:03.239430    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:37:03.257114 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:03.257083    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:37:03.270703 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:03.270670    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:37:20.324954 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.324914    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"]
Nov 01 12:37:20.343404 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.343362    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.385704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.382883    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"]
Nov 01 12:37:20.451073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.438467    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"openshift-service-ca.crt\""
Nov 01 12:37:20.451073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.438659    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"default-dockercfg-vp2p9\""
Nov 01 12:37:20.451073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.447278    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"premium-w1-5b1bd4-kserve-self-signed-certs\""
Nov 01 12:37:20.451073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.447451    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"kube-root-ca.crt\""
Nov 01 12:37:20.484920 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.484882    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-tmp-dir\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.485299 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.485274    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d7nn\" (UniqueName: \"kubernetes.io/projected/72055ebb-472b-43c6-8d29-e7a870069f31-kube-api-access-9d7nn\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.485444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.485427    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-dshm\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.497360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.497327    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-home\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.506192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.506155    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-model-cache\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.506681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.506654    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72055ebb-472b-43c6-8d29-e7a870069f31-tls-certs\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.611109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.608537    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72055ebb-472b-43c6-8d29-e7a870069f31-tls-certs\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.611109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.608598    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-tmp-dir\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.611109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.608684    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d7nn\" (UniqueName: \"kubernetes.io/projected/72055ebb-472b-43c6-8d29-e7a870069f31-kube-api-access-9d7nn\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.611109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.608710    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-dshm\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.611109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.608806    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-home\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.611109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.608834    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-model-cache\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.611109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.609353    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-model-cache\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.611109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.609615    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-tmp-dir\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.611109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.610561    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-home\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.612348 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.612320    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-dshm\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.612974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.612954    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72055ebb-472b-43c6-8d29-e7a870069f31-tls-certs\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.671089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.671031    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d7nn\" (UniqueName: \"kubernetes.io/projected/72055ebb-472b-43c6-8d29-e7a870069f31-kube-api-access-9d7nn\") pod \"premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") " pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:20.706774 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:20.704508    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:21.144092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:21.144041    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"]
Nov 01 12:37:21.151141 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:37:21.151107    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72055ebb_472b_43c6_8d29_e7a870069f31.slice/crio-4687da12ca171195d7bf8d805bc0afed609a7c5145cdf6c72bc0a24b81770a42 WatchSource:0}: Error finding container 4687da12ca171195d7bf8d805bc0afed609a7c5145cdf6c72bc0a24b81770a42: Status 404 returned error can't find the container with id 4687da12ca171195d7bf8d805bc0afed609a7c5145cdf6c72bc0a24b81770a42
Nov 01 12:37:21.419640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:21.419546    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" event={"ID":"72055ebb-472b-43c6-8d29-e7a870069f31","Type":"ContainerStarted","Data":"4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730"}
Nov 01 12:37:21.419640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:21.419592    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" event={"ID":"72055ebb-472b-43c6-8d29-e7a870069f31","Type":"ContainerStarted","Data":"4687da12ca171195d7bf8d805bc0afed609a7c5145cdf6c72bc0a24b81770a42"}
Nov 01 12:37:21.528799 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:21.528740    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podStartSLOduration=1.528723734 podStartE2EDuration="1.528723734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:37:21.5275286 +0000 UTC m=+883.034719837" watchObservedRunningTime="2026-11-01 12:37:21.528723734 +0000 UTC m=+883.035914994"
Nov 01 12:37:21.705161 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:21.705073    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:30.705463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:30.705416    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:30.718044 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:30.718011    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:31.027940 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:37:31.027903    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:37:31.502734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:31.502703    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:37:38.832728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:38.832695    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:37:38.832728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:38.832720    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:37:38.842231 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:38.842199    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:37:38.842391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:38.842379    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:37:55.985772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:55.985671    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"]
Nov 01 12:37:55.989994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:55.989971    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.043200 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.043166    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"]
Nov 01 12:37:56.085984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.085932    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:37:56.085984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.085934    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:37:56.112677 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.112641    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2zh\" (UniqueName: \"kubernetes.io/projected/e2303129-8a80-4d6b-83cd-6ca548c8ca43-kube-api-access-8d2zh\") pod \"maas-api-revoke-sub-ea0a74ae-xxm8w\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.112886 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.112717    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea0a74ae-xxm8w\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.112960 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.112907    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea0a74ae-xxm8w\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.214184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.214143    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2zh\" (UniqueName: \"kubernetes.io/projected/e2303129-8a80-4d6b-83cd-6ca548c8ca43-kube-api-access-8d2zh\") pod \"maas-api-revoke-sub-ea0a74ae-xxm8w\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.214391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.214207    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea0a74ae-xxm8w\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.214391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.214291    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea0a74ae-xxm8w\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.214979 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.214949    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea0a74ae-xxm8w\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.216760 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.216727    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea0a74ae-xxm8w\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.268858 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.268771    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2zh\" (UniqueName: \"kubernetes.io/projected/e2303129-8a80-4d6b-83cd-6ca548c8ca43-kube-api-access-8d2zh\") pod \"maas-api-revoke-sub-ea0a74ae-xxm8w\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.305344 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.305311    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:37:56.680271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.680190    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"]
Nov 01 12:37:56.686577 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:37:56.686549    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2303129_8a80_4d6b_83cd_6ca548c8ca43.slice/crio-b8a9bb85bc971b45215f06f1eab88392c69e8dfcb7bddf0fbd69d3f522f7c207 WatchSource:0}: Error finding container b8a9bb85bc971b45215f06f1eab88392c69e8dfcb7bddf0fbd69d3f522f7c207: Status 404 returned error can't find the container with id b8a9bb85bc971b45215f06f1eab88392c69e8dfcb7bddf0fbd69d3f522f7c207
Nov 01 12:37:56.695502 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:37:56.695472    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w" event={"ID":"e2303129-8a80-4d6b-83cd-6ca548c8ca43","Type":"ContainerStarted","Data":"b8a9bb85bc971b45215f06f1eab88392c69e8dfcb7bddf0fbd69d3f522f7c207"}
Nov 01 12:38:01.027123 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:38:01.027078    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:38:01.749940 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:01.749908    5725 generic.go:356] "Generic (PLEG): container finished" podID="e2303129-8a80-4d6b-83cd-6ca548c8ca43" containerID="7a0b72c34a4d8d07787947236edd634fade0a176ee6bedfe04b79518b03dd07f" exitCode=0
Nov 01 12:38:01.750236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:01.749951    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w" event={"ID":"e2303129-8a80-4d6b-83cd-6ca548c8ca43","Type":"ContainerDied","Data":"7a0b72c34a4d8d07787947236edd634fade0a176ee6bedfe04b79518b03dd07f"}
Nov 01 12:38:02.915844 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:02.915814    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:38:02.998293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:02.998254    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-xks-ca\") pod \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") "
Nov 01 12:38:02.998293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:02.998299    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-openshift-ca\") pod \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") "
Nov 01 12:38:02.998538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:02.998446    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2303129-8a80-4d6b-83cd-6ca548c8ca43-kube-api-access-8d2zh\" (UniqueName: \"kubernetes.io/projected/e2303129-8a80-4d6b-83cd-6ca548c8ca43-kube-api-access-8d2zh\") pod \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\" (UID: \"e2303129-8a80-4d6b-83cd-6ca548c8ca43\") "
Nov 01 12:38:02.998747 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:02.998722    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-openshift-ca" pod "e2303129-8a80-4d6b-83cd-6ca548c8ca43" (UID: "e2303129-8a80-4d6b-83cd-6ca548c8ca43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:02.998898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:02.998882    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:03.000704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:03.000677    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-xks-ca" pod "e2303129-8a80-4d6b-83cd-6ca548c8ca43" (UID: "e2303129-8a80-4d6b-83cd-6ca548c8ca43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:38:03.000820 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:03.000756    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2303129-8a80-4d6b-83cd-6ca548c8ca43-kube-api-access-8d2zh" pod "e2303129-8a80-4d6b-83cd-6ca548c8ca43" (UID: "e2303129-8a80-4d6b-83cd-6ca548c8ca43"). InnerVolumeSpecName "kube-api-access-8d2zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:03.100261 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:03.100223    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2303129-8a80-4d6b-83cd-6ca548c8ca43-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:03.100261 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:03.100254    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2zh\" (UniqueName: \"kubernetes.io/projected/e2303129-8a80-4d6b-83cd-6ca548c8ca43-kube-api-access-8d2zh\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:03.765859 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:03.765827    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"
Nov 01 12:38:03.766081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:03.765864    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w" event={"ID":"e2303129-8a80-4d6b-83cd-6ca548c8ca43","Type":"ContainerDied","Data":"b8a9bb85bc971b45215f06f1eab88392c69e8dfcb7bddf0fbd69d3f522f7c207"}
Nov 01 12:38:03.766081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:03.765924    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8a9bb85bc971b45215f06f1eab88392c69e8dfcb7bddf0fbd69d3f522f7c207"
Nov 01 12:38:11.325203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.325169    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"]
Nov 01 12:38:11.325702 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.325689    5725 cpu_manager.go:414] "containerMap: removing container" podUID="e2303129-8a80-4d6b-83cd-6ca548c8ca43" containerName="revoke-keys"
Nov 01 12:38:11.325702 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.325703    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2303129-8a80-4d6b-83cd-6ca548c8ca43" containerName="revoke-keys"
Nov 01 12:38:11.325787 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.325778    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2303129-8a80-4d6b-83cd-6ca548c8ca43" containerName="revoke-keys"
Nov 01 12:38:11.329310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.329290    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.381739 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.381707    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"]
Nov 01 12:38:11.395328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.395290    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a04b2e-sv7np\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.395481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.395334    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jpfg\" (UniqueName: \"kubernetes.io/projected/48d063d9-24ce-48bc-b915-4e21b494baba-kube-api-access-7jpfg\") pod \"maas-api-revoke-sub-51a04b2e-sv7np\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.395481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.395438    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a04b2e-sv7np\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.424537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.424504    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:38:11.424707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.424563    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:38:11.496351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.496297    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a04b2e-sv7np\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.496530 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.496474    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a04b2e-sv7np\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.496530 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.496500    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jpfg\" (UniqueName: \"kubernetes.io/projected/48d063d9-24ce-48bc-b915-4e21b494baba-kube-api-access-7jpfg\") pod \"maas-api-revoke-sub-51a04b2e-sv7np\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.497010 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.496988    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a04b2e-sv7np\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.498921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.498904    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a04b2e-sv7np\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.550275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.550240    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jpfg\" (UniqueName: \"kubernetes.io/projected/48d063d9-24ce-48bc-b915-4e21b494baba-kube-api-access-7jpfg\") pod \"maas-api-revoke-sub-51a04b2e-sv7np\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:11.643633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:11.643550    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:12.021325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:12.021290    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"]
Nov 01 12:38:12.028170 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:38:12.028142    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d063d9_24ce_48bc_b915_4e21b494baba.slice/crio-2afe68bcfd4fb7a463f5d34b9fd71a80c42ff46a9132f1364492f139ac9d6c2d WatchSource:0}: Error finding container 2afe68bcfd4fb7a463f5d34b9fd71a80c42ff46a9132f1364492f139ac9d6c2d: Status 404 returned error can't find the container with id 2afe68bcfd4fb7a463f5d34b9fd71a80c42ff46a9132f1364492f139ac9d6c2d
Nov 01 12:38:12.831246 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:12.831210    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np" event={"ID":"48d063d9-24ce-48bc-b915-4e21b494baba","Type":"ContainerStarted","Data":"b24e0a6d1fedc095d5e502b6459c15b14e9540f0f294a9a9475dc742530215f7"}
Nov 01 12:38:12.831727 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:12.831265    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np" event={"ID":"48d063d9-24ce-48bc-b915-4e21b494baba","Type":"ContainerStarted","Data":"2afe68bcfd4fb7a463f5d34b9fd71a80c42ff46a9132f1364492f139ac9d6c2d"}
Nov 01 12:38:12.940885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:12.940829    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np" podStartSLOduration=1.940810879 podStartE2EDuration="1.940810879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:38:12.938489383 +0000 UTC m=+934.445680601" watchObservedRunningTime="2026-11-01 12:38:12.940810879 +0000 UTC m=+934.448002099"
Nov 01 12:38:13.839565 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:13.839527    5725 generic.go:356] "Generic (PLEG): container finished" podID="48d063d9-24ce-48bc-b915-4e21b494baba" containerID="b24e0a6d1fedc095d5e502b6459c15b14e9540f0f294a9a9475dc742530215f7" exitCode=0
Nov 01 12:38:13.839966 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:13.839580    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np" event={"ID":"48d063d9-24ce-48bc-b915-4e21b494baba","Type":"ContainerDied","Data":"b24e0a6d1fedc095d5e502b6459c15b14e9540f0f294a9a9475dc742530215f7"}
Nov 01 12:38:14.328519 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.328479    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"]
Nov 01 12:38:14.332811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.332791    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.386260 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.386211    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"]
Nov 01 12:38:14.426914 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.426878    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-dshm\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.427130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.426928    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92zp4\" (UniqueName: \"kubernetes.io/projected/0decb764-723c-4026-95d9-8096aabf3571-kube-api-access-92zp4\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.427130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.427026    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0decb764-723c-4026-95d9-8096aabf3571-tls-certs\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.427276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.427168    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-model-cache\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.427276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.427206    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-home\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.427380 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.427315    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-tmp-dir\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.429261 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.429236    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"premium-w5-9514b0-kserve-self-signed-certs\""
Nov 01 12:38:14.528346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.528300    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-tmp-dir\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.528515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.528386    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-dshm\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.528515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.528417    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92zp4\" (UniqueName: \"kubernetes.io/projected/0decb764-723c-4026-95d9-8096aabf3571-kube-api-access-92zp4\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.528515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.528450    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0decb764-723c-4026-95d9-8096aabf3571-tls-certs\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.528515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.528499    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-model-cache\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.528515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.528515    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-home\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.528804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.528767    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-tmp-dir\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.528866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.528797    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-home\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.529390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.529367    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-model-cache\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.531732 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.531711    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0decb764-723c-4026-95d9-8096aabf3571-tls-certs\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.531889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.531875    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-dshm\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.586555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.586482    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92zp4\" (UniqueName: \"kubernetes.io/projected/0decb764-723c-4026-95d9-8096aabf3571-kube-api-access-92zp4\") pod \"premium-w5-9514b0-kserve-5cf5658fbb-rjxkf\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") " pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.646268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.646230    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:14.978143 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:14.978118    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:15.036106 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.036036    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d063d9-24ce-48bc-b915-4e21b494baba-kube-api-access-7jpfg\" (UniqueName: \"kubernetes.io/projected/48d063d9-24ce-48bc-b915-4e21b494baba-kube-api-access-7jpfg\") pod \"48d063d9-24ce-48bc-b915-4e21b494baba\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") "
Nov 01 12:38:15.036303 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.036170    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-openshift-ca\") pod \"48d063d9-24ce-48bc-b915-4e21b494baba\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") "
Nov 01 12:38:15.036303 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.036201    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-xks-ca\") pod \"48d063d9-24ce-48bc-b915-4e21b494baba\" (UID: \"48d063d9-24ce-48bc-b915-4e21b494baba\") "
Nov 01 12:38:15.036584 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.036549    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-openshift-ca" pod "48d063d9-24ce-48bc-b915-4e21b494baba" (UID: "48d063d9-24ce-48bc-b915-4e21b494baba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:15.036714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.036680    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:15.038191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.038167    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-xks-ca" pod "48d063d9-24ce-48bc-b915-4e21b494baba" (UID: "48d063d9-24ce-48bc-b915-4e21b494baba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:38:15.038275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.038236    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d063d9-24ce-48bc-b915-4e21b494baba-kube-api-access-7jpfg" pod "48d063d9-24ce-48bc-b915-4e21b494baba" (UID: "48d063d9-24ce-48bc-b915-4e21b494baba"). InnerVolumeSpecName "kube-api-access-7jpfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:15.075133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.075100    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"]
Nov 01 12:38:15.081944 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:38:15.081920    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0decb764_723c_4026_95d9_8096aabf3571.slice/crio-3c29c77408f64cdef45bb799e547100db6035787767eb11ea137851b85d325a3 WatchSource:0}: Error finding container 3c29c77408f64cdef45bb799e547100db6035787767eb11ea137851b85d325a3: Status 404 returned error can't find the container with id 3c29c77408f64cdef45bb799e547100db6035787767eb11ea137851b85d325a3
Nov 01 12:38:15.138042 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.137994    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d063d9-24ce-48bc-b915-4e21b494baba-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:15.138042 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.138015    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jpfg\" (UniqueName: \"kubernetes.io/projected/48d063d9-24ce-48bc-b915-4e21b494baba-kube-api-access-7jpfg\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:15.856634 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.856540    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" event={"ID":"0decb764-723c-4026-95d9-8096aabf3571","Type":"ContainerStarted","Data":"6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e"}
Nov 01 12:38:15.856634 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.856588    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" event={"ID":"0decb764-723c-4026-95d9-8096aabf3571","Type":"ContainerStarted","Data":"3c29c77408f64cdef45bb799e547100db6035787767eb11ea137851b85d325a3"}
Nov 01 12:38:15.858185 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.858160    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"
Nov 01 12:38:15.858326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.858201    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np" event={"ID":"48d063d9-24ce-48bc-b915-4e21b494baba","Type":"ContainerDied","Data":"2afe68bcfd4fb7a463f5d34b9fd71a80c42ff46a9132f1364492f139ac9d6c2d"}
Nov 01 12:38:15.858326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.858249    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2afe68bcfd4fb7a463f5d34b9fd71a80c42ff46a9132f1364492f139ac9d6c2d"
Nov 01 12:38:15.968322 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:15.968259    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podStartSLOduration=1.968239535 podStartE2EDuration="1.968239535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:38:15.963975042 +0000 UTC m=+937.471166260" watchObservedRunningTime="2026-11-01 12:38:15.968239535 +0000 UTC m=+937.475430753"
Nov 01 12:38:16.647210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:16.647177    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:18.451157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.451121    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"]
Nov 01 12:38:18.451659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.451645    5725 cpu_manager.go:414] "containerMap: removing container" podUID="48d063d9-24ce-48bc-b915-4e21b494baba" containerName="revoke-keys"
Nov 01 12:38:18.451704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.451660    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d063d9-24ce-48bc-b915-4e21b494baba" containerName="revoke-keys"
Nov 01 12:38:18.451784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.451772    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d063d9-24ce-48bc-b915-4e21b494baba" containerName="revoke-keys"
Nov 01 12:38:18.457744 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.457725    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.508223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.508188    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"]
Nov 01 12:38:18.550968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.550932    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:38:18.550968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.550960    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:38:18.578950 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.578909    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0310f236-fsvll\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.579186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.578975    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0310f236-fsvll\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.579186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.579146    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjlhz\" (UniqueName: \"kubernetes.io/projected/6ea5ad83-0931-4ed3-86d2-212132864725-kube-api-access-cjlhz\") pod \"maas-api-revoke-sub-0310f236-fsvll\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.680137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.680093    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0310f236-fsvll\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.680345 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.680253    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0310f236-fsvll\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.680439 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.680420    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlhz\" (UniqueName: \"kubernetes.io/projected/6ea5ad83-0931-4ed3-86d2-212132864725-kube-api-access-cjlhz\") pod \"maas-api-revoke-sub-0310f236-fsvll\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.681099 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.681034    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0310f236-fsvll\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.682466 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.682445    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0310f236-fsvll\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.734759 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.734731    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlhz\" (UniqueName: \"kubernetes.io/projected/6ea5ad83-0931-4ed3-86d2-212132864725-kube-api-access-cjlhz\") pod \"maas-api-revoke-sub-0310f236-fsvll\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:18.772556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:18.772520    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:19.155660 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:19.155566    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"]
Nov 01 12:38:19.893141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:19.892746    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll" event={"ID":"6ea5ad83-0931-4ed3-86d2-212132864725","Type":"ContainerStarted","Data":"7451c010d7b64e96653c1916af018bd887803e085e93322a2e585e7c4aa03b60"}
Nov 01 12:38:19.893141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:19.892805    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll" event={"ID":"6ea5ad83-0931-4ed3-86d2-212132864725","Type":"ContainerStarted","Data":"f14658d11580d429b67a5d1b32dc45f99ce22fa972ea043f7ac7931f370ea1a9"}
Nov 01 12:38:20.004233 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:20.004167    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll" podStartSLOduration=2.004147882 podStartE2EDuration="2.004147882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:38:19.999236748 +0000 UTC m=+941.506427978" watchObservedRunningTime="2026-11-01 12:38:20.004147882 +0000 UTC m=+941.511339101"
Nov 01 12:38:20.902256 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:20.902220    5725 generic.go:356] "Generic (PLEG): container finished" podID="6ea5ad83-0931-4ed3-86d2-212132864725" containerID="7451c010d7b64e96653c1916af018bd887803e085e93322a2e585e7c4aa03b60" exitCode=0
Nov 01 12:38:20.902659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:20.902283    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll" event={"ID":"6ea5ad83-0931-4ed3-86d2-212132864725","Type":"ContainerDied","Data":"7451c010d7b64e96653c1916af018bd887803e085e93322a2e585e7c4aa03b60"}
Nov 01 12:38:22.068998 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.068971    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:22.220602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.220566    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea5ad83-0931-4ed3-86d2-212132864725-kube-api-access-cjlhz\" (UniqueName: \"kubernetes.io/projected/6ea5ad83-0931-4ed3-86d2-212132864725-kube-api-access-cjlhz\") pod \"6ea5ad83-0931-4ed3-86d2-212132864725\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") "
Nov 01 12:38:22.220771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.220677    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-xks-ca\") pod \"6ea5ad83-0931-4ed3-86d2-212132864725\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") "
Nov 01 12:38:22.220771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.220735    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-openshift-ca\") pod \"6ea5ad83-0931-4ed3-86d2-212132864725\" (UID: \"6ea5ad83-0931-4ed3-86d2-212132864725\") "
Nov 01 12:38:22.221193 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.221160    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-openshift-ca" pod "6ea5ad83-0931-4ed3-86d2-212132864725" (UID: "6ea5ad83-0931-4ed3-86d2-212132864725"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:22.221337 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.221278    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:22.222927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.222896    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea5ad83-0931-4ed3-86d2-212132864725-kube-api-access-cjlhz" pod "6ea5ad83-0931-4ed3-86d2-212132864725" (UID: "6ea5ad83-0931-4ed3-86d2-212132864725"). InnerVolumeSpecName "kube-api-access-cjlhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:22.225186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.225147    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-xks-ca" pod "6ea5ad83-0931-4ed3-86d2-212132864725" (UID: "6ea5ad83-0931-4ed3-86d2-212132864725"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:38:22.322296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.322259    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjlhz\" (UniqueName: \"kubernetes.io/projected/6ea5ad83-0931-4ed3-86d2-212132864725-kube-api-access-cjlhz\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:22.322296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.322290    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ea5ad83-0931-4ed3-86d2-212132864725-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:22.917777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.917697    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"
Nov 01 12:38:22.917777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.917711    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll" event={"ID":"6ea5ad83-0931-4ed3-86d2-212132864725","Type":"ContainerDied","Data":"f14658d11580d429b67a5d1b32dc45f99ce22fa972ea043f7ac7931f370ea1a9"}
Nov 01 12:38:22.917777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:22.917754    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f14658d11580d429b67a5d1b32dc45f99ce22fa972ea043f7ac7931f370ea1a9"
Nov 01 12:38:24.647065 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:24.647013    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:24.661903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:24.661871    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:24.941827 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:24.941736    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:38:26.906826 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:26.906794    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"]
Nov 01 12:38:26.907335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:26.907321    5725 cpu_manager.go:414] "containerMap: removing container" podUID="6ea5ad83-0931-4ed3-86d2-212132864725" containerName="revoke-keys"
Nov 01 12:38:26.907335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:26.907335    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea5ad83-0931-4ed3-86d2-212132864725" containerName="revoke-keys"
Nov 01 12:38:26.907434 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:26.907425    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea5ad83-0931-4ed3-86d2-212132864725" containerName="revoke-keys"
Nov 01 12:38:26.910529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:26.910512    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:26.968685 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:26.968646    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"]
Nov 01 12:38:27.012103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.012037    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:38:27.012103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.012037    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:38:27.077206 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.077174    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eda7ad1-pr9z2\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.077206 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.077215    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eda7ad1-pr9z2\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.077440 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.077408    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9f2\" (UniqueName: \"kubernetes.io/projected/34016853-e091-4d88-a644-a8b2aab4935d-kube-api-access-fs9f2\") pod \"maas-api-revoke-sub-1eda7ad1-pr9z2\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.188370 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.188273    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eda7ad1-pr9z2\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.188644 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.188620    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eda7ad1-pr9z2\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.188890 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.188873    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9f2\" (UniqueName: \"kubernetes.io/projected/34016853-e091-4d88-a644-a8b2aab4935d-kube-api-access-fs9f2\") pod \"maas-api-revoke-sub-1eda7ad1-pr9z2\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.201070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.190249    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eda7ad1-pr9z2\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.221672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.221598    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eda7ad1-pr9z2\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.242917 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.242871    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9f2\" (UniqueName: \"kubernetes.io/projected/34016853-e091-4d88-a644-a8b2aab4935d-kube-api-access-fs9f2\") pod \"maas-api-revoke-sub-1eda7ad1-pr9z2\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.523038 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.523001    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:27.899591 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.899506    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"]
Nov 01 12:38:27.911666 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:38:27.911630    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34016853_e091_4d88_a644_a8b2aab4935d.slice/crio-c7d6787ad6d2f94646bbb239002cecfbd27b6d2d46b3a97df4a50fd6c0eeb4d6 WatchSource:0}: Error finding container c7d6787ad6d2f94646bbb239002cecfbd27b6d2d46b3a97df4a50fd6c0eeb4d6: Status 404 returned error can't find the container with id c7d6787ad6d2f94646bbb239002cecfbd27b6d2d46b3a97df4a50fd6c0eeb4d6
Nov 01 12:38:27.952380 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:27.952350    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2" event={"ID":"34016853-e091-4d88-a644-a8b2aab4935d","Type":"ContainerStarted","Data":"c7d6787ad6d2f94646bbb239002cecfbd27b6d2d46b3a97df4a50fd6c0eeb4d6"}
Nov 01 12:38:28.959722 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:28.959689    5725 generic.go:356] "Generic (PLEG): container finished" podID="34016853-e091-4d88-a644-a8b2aab4935d" containerID="8fd851b3e6fe57c8c87ee038e8c5870f5c8d6935148d77e542d5d1ff2398b4dd" exitCode=0
Nov 01 12:38:28.960142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:28.959779    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2" event={"ID":"34016853-e091-4d88-a644-a8b2aab4935d","Type":"ContainerDied","Data":"8fd851b3e6fe57c8c87ee038e8c5870f5c8d6935148d77e542d5d1ff2398b4dd"}
Nov 01 12:38:29.182916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.182832    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"]
Nov 01 12:38:29.186654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.186634    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.199193 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.199166    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"]
Nov 01 12:38:29.203077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.203040    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.238766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.238730    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-94c94b-openshift-default-dockercfg-r5knk\""
Nov 01 12:38:29.243507 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.243474    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"]
Nov 01 12:38:29.307215 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.307176    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"]
Nov 01 12:38:29.312035 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.311999    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.312210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312077    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmz8n\" (UniqueName: \"kubernetes.io/projected/8294a4c4-14f9-4713-924d-7c7a1528f597-kube-api-access-mmz8n\") pod \"maas-api-revoke-sub-0b26e86f-pdz9k\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.312210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312169    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.312210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312197    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.312394 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312222    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b26e86f-pdz9k\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.312394 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312272    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.312394 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312353    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30185b0c-125b-45f0-b1fa-14af84e9a913-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.312551 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312416    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.312551 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312443    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.312551 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312473    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.312707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312573    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgx57\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-kube-api-access-kgx57\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.312707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.312608    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b26e86f-pdz9k\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.413290 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413257    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.413509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413303    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmz8n\" (UniqueName: \"kubernetes.io/projected/8294a4c4-14f9-4713-924d-7c7a1528f597-kube-api-access-mmz8n\") pod \"maas-api-revoke-sub-0b26e86f-pdz9k\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.413509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413333    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.413695 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413564    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.413695 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413604    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b26e86f-pdz9k\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.413695 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413634    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.413695 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413660    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30185b0c-125b-45f0-b1fa-14af84e9a913-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.413905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413712    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.413905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413754    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.413905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413806    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.413905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413865    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgx57\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-kube-api-access-kgx57\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.413905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.413897    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b26e86f-pdz9k\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.414196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.414030    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.414250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.414193    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.414350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.414323    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.414487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.414445    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30185b0c-125b-45f0-b1fa-14af84e9a913-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.414487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.414404    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b26e86f-pdz9k\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.414700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.414676    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.415953 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.415933    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.416132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.416114    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.416286 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.416270    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b26e86f-pdz9k\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.467565 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.467489    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgx57\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-kube-api-access-kgx57\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.467703 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.467592    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmz8n\" (UniqueName: \"kubernetes.io/projected/8294a4c4-14f9-4713-924d-7c7a1528f597-kube-api-access-mmz8n\") pod \"maas-api-revoke-sub-0b26e86f-pdz9k\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.467703 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.467600    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.500983 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.500948    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:29.516115 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.516091    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:29.938966 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.938932    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"]
Nov 01 12:38:29.943650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.943617    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"]
Nov 01 12:38:29.944360 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:38:29.944337    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8294a4c4_14f9_4713_924d_7c7a1528f597.slice/crio-5c05048ab235eae3f36049b17cdad5db54b5641fd69893e27959db77f7ff940b WatchSource:0}: Error finding container 5c05048ab235eae3f36049b17cdad5db54b5641fd69893e27959db77f7ff940b: Status 404 returned error can't find the container with id 5c05048ab235eae3f36049b17cdad5db54b5641fd69893e27959db77f7ff940b
Nov 01 12:38:29.946780 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:38:29.946761    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30185b0c_125b_45f0_b1fa_14af84e9a913.slice/crio-665d49c98a2fd20f1a300cfd75e6817b83938d19888e8588e603ee5afcb54d2d WatchSource:0}: Error finding container 665d49c98a2fd20f1a300cfd75e6817b83938d19888e8588e603ee5afcb54d2d: Status 404 returned error can't find the container with id 665d49c98a2fd20f1a300cfd75e6817b83938d19888e8588e603ee5afcb54d2d
Nov 01 12:38:29.949092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.949044    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:38:29.949158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.949122    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:38:29.949158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.949150    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:38:29.968486 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.968456    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" event={"ID":"30185b0c-125b-45f0-b1fa-14af84e9a913","Type":"ContainerStarted","Data":"665d49c98a2fd20f1a300cfd75e6817b83938d19888e8588e603ee5afcb54d2d"}
Nov 01 12:38:29.970441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:29.970411    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k" event={"ID":"8294a4c4-14f9-4713-924d-7c7a1528f597","Type":"ContainerStarted","Data":"5c05048ab235eae3f36049b17cdad5db54b5641fd69893e27959db77f7ff940b"}
Nov 01 12:38:30.138253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.138227    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:30.323578 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.323545    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34016853-e091-4d88-a644-a8b2aab4935d-kube-api-access-fs9f2\" (UniqueName: \"kubernetes.io/projected/34016853-e091-4d88-a644-a8b2aab4935d-kube-api-access-fs9f2\") pod \"34016853-e091-4d88-a644-a8b2aab4935d\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") "
Nov 01 12:38:30.323773 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.323627    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-xks-ca\") pod \"34016853-e091-4d88-a644-a8b2aab4935d\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") "
Nov 01 12:38:30.323773 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.323721    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-openshift-ca\") pod \"34016853-e091-4d88-a644-a8b2aab4935d\" (UID: \"34016853-e091-4d88-a644-a8b2aab4935d\") "
Nov 01 12:38:30.324137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.324108    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-openshift-ca" pod "34016853-e091-4d88-a644-a8b2aab4935d" (UID: "34016853-e091-4d88-a644-a8b2aab4935d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:30.325832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.325804    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34016853-e091-4d88-a644-a8b2aab4935d-kube-api-access-fs9f2" pod "34016853-e091-4d88-a644-a8b2aab4935d" (UID: "34016853-e091-4d88-a644-a8b2aab4935d"). InnerVolumeSpecName "kube-api-access-fs9f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:30.325832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.325827    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-xks-ca" pod "34016853-e091-4d88-a644-a8b2aab4935d" (UID: "34016853-e091-4d88-a644-a8b2aab4935d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:38:30.425162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.425124    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:30.425162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.425162    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs9f2\" (UniqueName: \"kubernetes.io/projected/34016853-e091-4d88-a644-a8b2aab4935d-kube-api-access-fs9f2\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:30.425401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.425178    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34016853-e091-4d88-a644-a8b2aab4935d-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:30.978648 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.978605    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" event={"ID":"30185b0c-125b-45f0-b1fa-14af84e9a913","Type":"ContainerStarted","Data":"027a9e92548b3613afc1a8138240b3173bd4c11dd0bfc9815a853a0ea17a4116"}
Nov 01 12:38:30.979958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.979938    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"
Nov 01 12:38:30.980100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.979957    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2" event={"ID":"34016853-e091-4d88-a644-a8b2aab4935d","Type":"ContainerDied","Data":"c7d6787ad6d2f94646bbb239002cecfbd27b6d2d46b3a97df4a50fd6c0eeb4d6"}
Nov 01 12:38:30.980100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.980001    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7d6787ad6d2f94646bbb239002cecfbd27b6d2d46b3a97df4a50fd6c0eeb4d6"
Nov 01 12:38:30.981686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:30.981663    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k" event={"ID":"8294a4c4-14f9-4713-924d-7c7a1528f597","Type":"ContainerStarted","Data":"e2019802688b028584dc0b2158be6331ce4c971b588cdc00c03831a54a60cee9"}
Nov 01 12:38:31.056869 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:38:31.056820    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:38:31.110851 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:31.110779    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podStartSLOduration=2.1107607919999998 podStartE2EDuration="2.110760792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:38:31.105371706 +0000 UTC m=+952.612562925" watchObservedRunningTime="2026-11-01 12:38:31.110760792 +0000 UTC m=+952.617952010"
Nov 01 12:38:31.213681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:31.213624    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k" podStartSLOduration=2.213610177 podStartE2EDuration="2.213610177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:38:31.212020718 +0000 UTC m=+952.719211937" watchObservedRunningTime="2026-11-01 12:38:31.213610177 +0000 UTC m=+952.720801393"
Nov 01 12:38:31.501202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:31.501163    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:31.502849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:31.502815    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:38:31.503003 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:31.502866    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 12:38:31.989661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:31.989624    5725 generic.go:356] "Generic (PLEG): container finished" podID="8294a4c4-14f9-4713-924d-7c7a1528f597" containerID="e2019802688b028584dc0b2158be6331ce4c971b588cdc00c03831a54a60cee9" exitCode=0
Nov 01 12:38:31.990079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:31.989711    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k" event={"ID":"8294a4c4-14f9-4713-924d-7c7a1528f597","Type":"ContainerDied","Data":"e2019802688b028584dc0b2158be6331ce4c971b588cdc00c03831a54a60cee9"}
Nov 01 12:38:32.502973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:32.502937    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:38:32.503163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:32.502989    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 12:38:33.203142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.203118    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:33.279512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.279421    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-openshift-ca\") pod \"8294a4c4-14f9-4713-924d-7c7a1528f597\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") "
Nov 01 12:38:33.279512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.279477    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8294a4c4-14f9-4713-924d-7c7a1528f597-kube-api-access-mmz8n\" (UniqueName: \"kubernetes.io/projected/8294a4c4-14f9-4713-924d-7c7a1528f597-kube-api-access-mmz8n\") pod \"8294a4c4-14f9-4713-924d-7c7a1528f597\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") "
Nov 01 12:38:33.279715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.279518    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-xks-ca\") pod \"8294a4c4-14f9-4713-924d-7c7a1528f597\" (UID: \"8294a4c4-14f9-4713-924d-7c7a1528f597\") "
Nov 01 12:38:33.279777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.279752    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-openshift-ca" pod "8294a4c4-14f9-4713-924d-7c7a1528f597" (UID: "8294a4c4-14f9-4713-924d-7c7a1528f597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:33.279909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.279894    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:33.281670 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.281639    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8294a4c4-14f9-4713-924d-7c7a1528f597-kube-api-access-mmz8n" pod "8294a4c4-14f9-4713-924d-7c7a1528f597" (UID: "8294a4c4-14f9-4713-924d-7c7a1528f597"). InnerVolumeSpecName "kube-api-access-mmz8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:33.281670 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.281643    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-xks-ca" pod "8294a4c4-14f9-4713-924d-7c7a1528f597" (UID: "8294a4c4-14f9-4713-924d-7c7a1528f597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:38:33.381579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.381534    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmz8n\" (UniqueName: \"kubernetes.io/projected/8294a4c4-14f9-4713-924d-7c7a1528f597-kube-api-access-mmz8n\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:33.381579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.381579    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8294a4c4-14f9-4713-924d-7c7a1528f597-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:33.501468 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.501432    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:38:33.501641 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:33.501486    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 12:38:34.002877 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:34.002838    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"
Nov 01 12:38:34.003071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:34.002907    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k" event={"ID":"8294a4c4-14f9-4713-924d-7c7a1528f597","Type":"ContainerDied","Data":"5c05048ab235eae3f36049b17cdad5db54b5641fd69893e27959db77f7ff940b"}
Nov 01 12:38:34.003071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:34.002954    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c05048ab235eae3f36049b17cdad5db54b5641fd69893e27959db77f7ff940b"
Nov 01 12:38:34.502796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:34.502759    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:38:34.503186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:34.502810    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 12:38:35.501964 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:35.501928    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:38:35.502190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:35.501980    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 12:38:36.502563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:36.502523    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:38:36.503099 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:36.502580    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 12:38:38.184583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:38.183365    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:38.184583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:38.184366    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:38.190564 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:38.190530    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:38:44.273778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:44.273734    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"]
Nov 01 12:38:44.274399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:44.273983    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy" containerID="cri-o://7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9" gracePeriod=30
Nov 01 12:38:44.320176 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:44.320141    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"]
Nov 01 12:38:44.320588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:44.320560    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy" containerID="cri-o://42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732" gracePeriod=30
Nov 01 12:38:44.324075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:44.324017    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"]
Nov 01 12:38:44.324292 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:44.324269    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerName="istio-proxy" containerID="cri-o://856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589" gracePeriod=30
Nov 01 12:38:49.619494 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.619460    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:38:49.670346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.670326    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:38:49.673823 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.673803    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:38:49.705676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705646    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-certs\") pod \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") "
Nov 01 12:38:49.705676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705679    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-token\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-token\") pod \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") "
Nov 01 12:38:49.705932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705698    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a99f825-ca55-4acf-97e4-10ea52ae71da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a99f825-ca55-4acf-97e4-10ea52ae71da-istiod-ca-cert\") pod \"4a99f825-ca55-4acf-97e4-10ea52ae71da\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") "
Nov 01 12:38:49.705932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705749    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-kube-api-access-p2nx8\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-kube-api-access-p2nx8\") pod \"9ec5523e-d2ad-414c-9186-003789159540\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") "
Nov 01 12:38:49.705932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705803    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-kube-api-access-zgf87\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-kube-api-access-zgf87\") pod \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") "
Nov 01 12:38:49.705932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705833    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec5523e-d2ad-414c-9186-003789159540-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec5523e-d2ad-414c-9186-003789159540-istiod-ca-cert\") pod \"9ec5523e-d2ad-414c-9186-003789159540\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") "
Nov 01 12:38:49.705932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705891    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istiod-ca-cert\") pod \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") "
Nov 01 12:38:49.705932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705918    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-socket\") pod \"4a99f825-ca55-4acf-97e4-10ea52ae71da\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") "
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705942    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-certs" pod "91ccefa7-cab2-4c4d-951e-1036ecaff01b" (UID: "91ccefa7-cab2-4c4d-951e-1036ecaff01b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.705963    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-podinfo\") pod \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") "
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706030    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-envoy\") pod \"9ec5523e-d2ad-414c-9186-003789159540\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") "
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706084    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-data\") pod \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") "
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706100    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a99f825-ca55-4acf-97e4-10ea52ae71da-istiod-ca-cert" pod "4a99f825-ca55-4acf-97e4-10ea52ae71da" (UID: "4a99f825-ca55-4acf-97e4-10ea52ae71da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706149    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-certs\") pod \"4a99f825-ca55-4acf-97e4-10ea52ae71da\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") "
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706178    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-data\") pod \"9ec5523e-d2ad-414c-9186-003789159540\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") "
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706202    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-socket\") pod \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") "
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706242    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-istio-token\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-istio-token\") pod \"9ec5523e-d2ad-414c-9186-003789159540\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") "
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706274    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-token\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-token\") pod \"4a99f825-ca55-4acf-97e4-10ea52ae71da\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") "
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706269    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-socket" pod "4a99f825-ca55-4acf-97e4-10ea52ae71da" (UID: "4a99f825-ca55-4acf-97e4-10ea52ae71da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.706300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706292    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-socket\") pod \"9ec5523e-d2ad-414c-9186-003789159540\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") "
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706336    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istiod-ca-cert" pod "91ccefa7-cab2-4c4d-951e-1036ecaff01b" (UID: "91ccefa7-cab2-4c4d-951e-1036ecaff01b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706342    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-certs\") pod \"9ec5523e-d2ad-414c-9186-003789159540\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") "
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706401    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-data\") pod \"4a99f825-ca55-4acf-97e4-10ea52ae71da\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") "
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706438    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-envoy\") pod \"4a99f825-ca55-4acf-97e4-10ea52ae71da\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") "
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706468    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-credential-socket\") pod \"9ec5523e-d2ad-414c-9186-003789159540\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") "
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706505    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-envoy\") pod \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") "
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706638    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-data" pod "9ec5523e-d2ad-414c-9186-003789159540" (UID: "9ec5523e-d2ad-414c-9186-003789159540"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706670    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-kube-api-access-hjwfq\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-kube-api-access-hjwfq\") pod \"4a99f825-ca55-4acf-97e4-10ea52ae71da\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") "
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706708    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-credential-socket\") pod \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\" (UID: \"91ccefa7-cab2-4c4d-951e-1036ecaff01b\") "
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706722    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec5523e-d2ad-414c-9186-003789159540-istiod-ca-cert" pod "9ec5523e-d2ad-414c-9186-003789159540" (UID: "9ec5523e-d2ad-414c-9186-003789159540"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706835    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-socket" pod "91ccefa7-cab2-4c4d-951e-1036ecaff01b" (UID: "91ccefa7-cab2-4c4d-951e-1036ecaff01b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706735    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-podinfo\") pod \"4a99f825-ca55-4acf-97e4-10ea52ae71da\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") "
Nov 01 12:38:49.706907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706894    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-credential-socket\") pod \"4a99f825-ca55-4acf-97e4-10ea52ae71da\" (UID: \"4a99f825-ca55-4acf-97e4-10ea52ae71da\") "
Nov 01 12:38:49.707807 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706959    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-data" pod "91ccefa7-cab2-4c4d-951e-1036ecaff01b" (UID: "91ccefa7-cab2-4c4d-951e-1036ecaff01b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.707807 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.706965    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ec5523e-d2ad-414c-9186-003789159540-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec5523e-d2ad-414c-9186-003789159540-istio-podinfo\") pod \"9ec5523e-d2ad-414c-9186-003789159540\" (UID: \"9ec5523e-d2ad-414c-9186-003789159540\") "
Nov 01 12:38:49.710006 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.709954    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-credential-socket" pod "9ec5523e-d2ad-414c-9186-003789159540" (UID: "9ec5523e-d2ad-414c-9186-003789159540"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.710406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.710185    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-envoy" pod "9ec5523e-d2ad-414c-9186-003789159540" (UID: "9ec5523e-d2ad-414c-9186-003789159540"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.710406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.710358    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-certs" pod "9ec5523e-d2ad-414c-9186-003789159540" (UID: "9ec5523e-d2ad-414c-9186-003789159540"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.710617 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.710519    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-credential-socket" pod "91ccefa7-cab2-4c4d-951e-1036ecaff01b" (UID: "91ccefa7-cab2-4c4d-951e-1036ecaff01b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.710759 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.710735    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-kube-api-access-p2nx8" pod "9ec5523e-d2ad-414c-9186-003789159540" (UID: "9ec5523e-d2ad-414c-9186-003789159540"). InnerVolumeSpecName "kube-api-access-p2nx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:49.711510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.711301    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ec5523e-d2ad-414c-9186-003789159540-istio-podinfo" pod "9ec5523e-d2ad-414c-9186-003789159540" (UID: "9ec5523e-d2ad-414c-9186-003789159540"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:38:49.711510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.711337    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-socket" pod "9ec5523e-d2ad-414c-9186-003789159540" (UID: "9ec5523e-d2ad-414c-9186-003789159540"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.711510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.711366    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-kube-api-access-zgf87" pod "91ccefa7-cab2-4c4d-951e-1036ecaff01b" (UID: "91ccefa7-cab2-4c4d-951e-1036ecaff01b"). InnerVolumeSpecName "kube-api-access-zgf87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:49.711510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.711370    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-envoy" pod "91ccefa7-cab2-4c4d-951e-1036ecaff01b" (UID: "91ccefa7-cab2-4c4d-951e-1036ecaff01b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.711510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.711435    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-token" pod "91ccefa7-cab2-4c4d-951e-1036ecaff01b" (UID: "91ccefa7-cab2-4c4d-951e-1036ecaff01b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:49.711510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.711487    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-data" pod "4a99f825-ca55-4acf-97e4-10ea52ae71da" (UID: "4a99f825-ca55-4acf-97e4-10ea52ae71da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.712305 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.711621    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-credential-socket" pod "4a99f825-ca55-4acf-97e4-10ea52ae71da" (UID: "4a99f825-ca55-4acf-97e4-10ea52ae71da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.712305 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.711706    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-podinfo" pod "91ccefa7-cab2-4c4d-951e-1036ecaff01b" (UID: "91ccefa7-cab2-4c4d-951e-1036ecaff01b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:38:49.712305 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.711738    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-certs" pod "4a99f825-ca55-4acf-97e4-10ea52ae71da" (UID: "4a99f825-ca55-4acf-97e4-10ea52ae71da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.712305 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.712074    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-podinfo" pod "4a99f825-ca55-4acf-97e4-10ea52ae71da" (UID: "4a99f825-ca55-4acf-97e4-10ea52ae71da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:38:49.713611 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.713282    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-envoy" pod "4a99f825-ca55-4acf-97e4-10ea52ae71da" (UID: "4a99f825-ca55-4acf-97e4-10ea52ae71da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:38:49.713716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.713647    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec5523e-d2ad-414c-9186-003789159540-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.713716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.713675    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.713716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.713690    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.713716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.713705    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.713921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.713719    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.713921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.713738    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.713921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.713751    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.713921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.713764    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a99f825-ca55-4acf-97e4-10ea52ae71da-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.722909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.722869    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-kube-api-access-hjwfq" pod "4a99f825-ca55-4acf-97e4-10ea52ae71da" (UID: "4a99f825-ca55-4acf-97e4-10ea52ae71da"). InnerVolumeSpecName "kube-api-access-hjwfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:49.723126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.722931    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-istio-token" pod "9ec5523e-d2ad-414c-9186-003789159540" (UID: "9ec5523e-d2ad-414c-9186-003789159540"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:49.723280 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.723257    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-token" pod "4a99f825-ca55-4acf-97e4-10ea52ae71da" (UID: "4a99f825-ca55-4acf-97e4-10ea52ae71da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:49.814681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814645    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814680    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814694    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814707    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814720    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814733    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814745    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814757    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814769    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814781    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec5523e-d2ad-414c-9186-003789159540-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814793    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814807    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjwfq\" (UniqueName: \"kubernetes.io/projected/4a99f825-ca55-4acf-97e4-10ea52ae71da-kube-api-access-hjwfq\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814822    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ccefa7-cab2-4c4d-951e-1036ecaff01b-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814834    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a99f825-ca55-4acf-97e4-10ea52ae71da-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814846    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a99f825-ca55-4acf-97e4-10ea52ae71da-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814857    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec5523e-d2ad-414c-9186-003789159540-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814870    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814883    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2nx8\" (UniqueName: \"kubernetes.io/projected/9ec5523e-d2ad-414c-9186-003789159540-kube-api-access-p2nx8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:49.814937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:49.814894    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgf87\" (UniqueName: \"kubernetes.io/projected/91ccefa7-cab2-4c4d-951e-1036ecaff01b-kube-api-access-zgf87\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:50.158636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.158598    5725 generic.go:356] "Generic (PLEG): container finished" podID="9ec5523e-d2ad-414c-9186-003789159540" containerID="42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732" exitCode=0
Nov 01 12:38:50.158822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.158665    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"
Nov 01 12:38:50.158822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.158698    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" event={"ID":"9ec5523e-d2ad-414c-9186-003789159540","Type":"ContainerDied","Data":"42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732"}
Nov 01 12:38:50.158822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.158794    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4" event={"ID":"9ec5523e-d2ad-414c-9186-003789159540","Type":"ContainerDied","Data":"439528ad00db5af49d0b4bdc9607d2fe468d87080c893f9c4a414759e549f0f8"}
Nov 01 12:38:50.158822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.158819    5725 scope.go:122] "RemoveContainer" containerID="42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732"
Nov 01 12:38:50.160237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.160213    5725 generic.go:356] "Generic (PLEG): container finished" podID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerID="7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9" exitCode=0
Nov 01 12:38:50.160340 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.160270    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"
Nov 01 12:38:50.160340 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.160295    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" event={"ID":"4a99f825-ca55-4acf-97e4-10ea52ae71da","Type":"ContainerDied","Data":"7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9"}
Nov 01 12:38:50.160439 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.160353    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24" event={"ID":"4a99f825-ca55-4acf-97e4-10ea52ae71da","Type":"ContainerDied","Data":"9fecaffddd20be1bba913e8baa979a114c688223e2729851c11514399c49e1ec"}
Nov 01 12:38:50.161807 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.161783    5725 generic.go:356] "Generic (PLEG): container finished" podID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerID="856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589" exitCode=0
Nov 01 12:38:50.161927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.161812    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" event={"ID":"91ccefa7-cab2-4c4d-951e-1036ecaff01b","Type":"ContainerDied","Data":"856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589"}
Nov 01 12:38:50.161927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.161849    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"
Nov 01 12:38:50.161927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.161862    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p" event={"ID":"91ccefa7-cab2-4c4d-951e-1036ecaff01b","Type":"ContainerDied","Data":"7040c21842db367ff9f4a06841f6610a78ea065fb78cb1a9740f34391c205bd6"}
Nov 01 12:38:50.171429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.171393    5725 scope.go:122] "RemoveContainer" containerID="f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339"
Nov 01 12:38:50.181142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.181110    5725 scope.go:122] "RemoveContainer" containerID="42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732"
Nov 01 12:38:50.181397 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:38:50.181371    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732\": container with ID starting with 42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732 not found: ID does not exist" containerID="42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732"
Nov 01 12:38:50.181491 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.181405    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732"} err="failed to get container status \"42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732\": rpc error: code = NotFound desc = could not find container \"42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732\": container with ID starting with 42f6dc22f920563365c9b810258af0900586e61f717c542b42547de559c61732 not found: ID does not exist"
Nov 01 12:38:50.181491 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.181430    5725 scope.go:122] "RemoveContainer" containerID="f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339"
Nov 01 12:38:50.181685 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:38:50.181669    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339\": container with ID starting with f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339 not found: ID does not exist" containerID="f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339"
Nov 01 12:38:50.181728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.181693    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339"} err="failed to get container status \"f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339\": rpc error: code = NotFound desc = could not find container \"f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339\": container with ID starting with f09ccd4fc84533fd70d9e663e8cb1a31131a551826cdd6a7a9b9ca7ee3850339 not found: ID does not exist"
Nov 01 12:38:50.181728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.181709    5725 scope.go:122] "RemoveContainer" containerID="7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9"
Nov 01 12:38:50.191216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.191184    5725 scope.go:122] "RemoveContainer" containerID="7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9"
Nov 01 12:38:50.191514 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:38:50.191497    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9\": container with ID starting with 7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9 not found: ID does not exist" containerID="7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9"
Nov 01 12:38:50.191580 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.191520    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9"} err="failed to get container status \"7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9\": rpc error: code = NotFound desc = could not find container \"7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9\": container with ID starting with 7ed04a5b7ab93694d0f9e94b851e2786f638d71915ddcfa606da1e2ddf8c16b9 not found: ID does not exist"
Nov 01 12:38:50.191580 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.191535    5725 scope.go:122] "RemoveContainer" containerID="856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589"
Nov 01 12:38:50.200368 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.200349    5725 scope.go:122] "RemoveContainer" containerID="856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589"
Nov 01 12:38:50.200638 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:38:50.200623    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589\": container with ID starting with 856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589 not found: ID does not exist" containerID="856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589"
Nov 01 12:38:50.200685 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.200646    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589"} err="failed to get container status \"856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589\": rpc error: code = NotFound desc = could not find container \"856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589\": container with ID starting with 856e6b4930e8507d302c20381ac43dc916d676819dfdf947060e7f8476f1b589 not found: ID does not exist"
Nov 01 12:38:50.314903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.314877    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"]
Nov 01 12:38:50.318460 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.318433    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-h6tl4"]
Nov 01 12:38:50.421012 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.420911    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"]
Nov 01 12:38:50.427248 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.427214    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-znm24"]
Nov 01 12:38:50.528334 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.528294    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"]
Nov 01 12:38:50.532437 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.532410    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4ns6p"]
Nov 01 12:38:50.617129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.617094    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" path="/var/lib/kubelet/pods/4a99f825-ca55-4acf-97e4-10ea52ae71da/volumes"
Nov 01 12:38:50.617571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.617557    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" path="/var/lib/kubelet/pods/91ccefa7-cab2-4c4d-951e-1036ecaff01b/volumes"
Nov 01 12:38:50.617947 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:50.617935    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec5523e-d2ad-414c-9186-003789159540" path="/var/lib/kubelet/pods/9ec5523e-d2ad-414c-9186-003789159540/volumes"
Nov 01 12:38:51.935720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.935689    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"]
Nov 01 12:38:51.936205 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936191    5725 cpu_manager.go:414] "containerMap: removing container" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy"
Nov 01 12:38:51.936257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936206    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy"
Nov 01 12:38:51.936257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936233    5725 cpu_manager.go:414] "containerMap: removing container" podUID="34016853-e091-4d88-a644-a8b2aab4935d" containerName="revoke-keys"
Nov 01 12:38:51.936257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936238    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34016853-e091-4d88-a644-a8b2aab4935d" containerName="revoke-keys"
Nov 01 12:38:51.936257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936247    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8294a4c4-14f9-4713-924d-7c7a1528f597" containerName="revoke-keys"
Nov 01 12:38:51.936257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936253    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8294a4c4-14f9-4713-924d-7c7a1528f597" containerName="revoke-keys"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936263    5725 cpu_manager.go:414] "containerMap: removing container" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936269    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936284    5725 cpu_manager.go:414] "containerMap: removing container" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936289    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936296    5725 cpu_manager.go:414] "containerMap: removing container" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerName="istio-proxy"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936301    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerName="istio-proxy"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936375    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="34016853-e091-4d88-a644-a8b2aab4935d" containerName="revoke-keys"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936387    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8294a4c4-14f9-4713-924d-7c7a1528f597" containerName="revoke-keys"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936398    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a99f825-ca55-4acf-97e4-10ea52ae71da" containerName="istio-proxy"
Nov 01 12:38:51.936414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936412    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy"
Nov 01 12:38:51.936737 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936420    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ccefa7-cab2-4c4d-951e-1036ecaff01b" containerName="istio-proxy"
Nov 01 12:38:51.936737 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.936429    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec5523e-d2ad-414c-9186-003789159540" containerName="istio-proxy"
Nov 01 12:38:51.941849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.941825    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:51.991277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.991236    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:38:51.991277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.991267    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:38:51.992674 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:51.992651    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"]
Nov 01 12:38:52.040035 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.040000    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01bc3687-l69nx\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.040211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.040145    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01bc3687-l69nx\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.040211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.040195    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsb95\" (UniqueName: \"kubernetes.io/projected/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-kube-api-access-gsb95\") pod \"maas-api-revoke-sub-01bc3687-l69nx\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.141370 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.141333    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01bc3687-l69nx\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.141559 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.141410    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01bc3687-l69nx\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.141559 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.141454    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsb95\" (UniqueName: \"kubernetes.io/projected/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-kube-api-access-gsb95\") pod \"maas-api-revoke-sub-01bc3687-l69nx\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.142192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.142158    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01bc3687-l69nx\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.143792 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.143764    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01bc3687-l69nx\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.195509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.195437    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsb95\" (UniqueName: \"kubernetes.io/projected/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-kube-api-access-gsb95\") pod \"maas-api-revoke-sub-01bc3687-l69nx\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.255700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.255662    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:52.638258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:52.638228    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"]
Nov 01 12:38:52.645964 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:38:52.645927    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc355f7_12e8_42a5_bf79_1b9ef6be7d96.slice/crio-bc423ca115b474649374d979baab793270f9a30821f60a2e8afb9322e5488b17 WatchSource:0}: Error finding container bc423ca115b474649374d979baab793270f9a30821f60a2e8afb9322e5488b17: Status 404 returned error can't find the container with id bc423ca115b474649374d979baab793270f9a30821f60a2e8afb9322e5488b17
Nov 01 12:38:53.189279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.189194    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx" event={"ID":"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96","Type":"ContainerStarted","Data":"0e1988ce89a9e2e4e67a6e8c7aa1559ffee207762cc328e358634664aa207aba"}
Nov 01 12:38:53.189279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.189268    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx" event={"ID":"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96","Type":"ContainerStarted","Data":"bc423ca115b474649374d979baab793270f9a30821f60a2e8afb9322e5488b17"}
Nov 01 12:38:53.297848 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.297795    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx" podStartSLOduration=2.297782737 podStartE2EDuration="2.297782737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:38:53.296652887 +0000 UTC m=+974.803844117" watchObservedRunningTime="2026-11-01 12:38:53.297782737 +0000 UTC m=+974.804973954"
Nov 01 12:38:53.362345 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.362263    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"]
Nov 01 12:38:53.366931 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.366904    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.416134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.416105    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"]
Nov 01 12:38:53.454874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.454785    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce01fd73-2pzbp\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.455061 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.454962    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce01fd73-2pzbp\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.455275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.455252    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9m4d\" (UniqueName: \"kubernetes.io/projected/b5d15fcb-133f-40dd-b873-929f1b71a460-kube-api-access-n9m4d\") pod \"maas-api-revoke-sub-ce01fd73-2pzbp\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.556399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.556361    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9m4d\" (UniqueName: \"kubernetes.io/projected/b5d15fcb-133f-40dd-b873-929f1b71a460-kube-api-access-n9m4d\") pod \"maas-api-revoke-sub-ce01fd73-2pzbp\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.556399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.556413    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce01fd73-2pzbp\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.556638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.556458    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce01fd73-2pzbp\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.557120 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.557098    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce01fd73-2pzbp\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.559199 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.559168    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce01fd73-2pzbp\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.609981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.609948    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9m4d\" (UniqueName: \"kubernetes.io/projected/b5d15fcb-133f-40dd-b873-929f1b71a460-kube-api-access-n9m4d\") pod \"maas-api-revoke-sub-ce01fd73-2pzbp\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:53.683485 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:53.683446    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:54.058716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:54.058631    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"]
Nov 01 12:38:54.065967 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:38:54.065940    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d15fcb_133f_40dd_b873_929f1b71a460.slice/crio-12f2b263de4ce45cdd550c4f1c27ccf0c3e30eb78ce958835694fa7d317192a7 WatchSource:0}: Error finding container 12f2b263de4ce45cdd550c4f1c27ccf0c3e30eb78ce958835694fa7d317192a7: Status 404 returned error can't find the container with id 12f2b263de4ce45cdd550c4f1c27ccf0c3e30eb78ce958835694fa7d317192a7
Nov 01 12:38:54.197895 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:54.197860    5725 generic.go:356] "Generic (PLEG): container finished" podID="8cc355f7-12e8-42a5-bf79-1b9ef6be7d96" containerID="0e1988ce89a9e2e4e67a6e8c7aa1559ffee207762cc328e358634664aa207aba" exitCode=0
Nov 01 12:38:54.198302 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:54.197941    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx" event={"ID":"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96","Type":"ContainerDied","Data":"0e1988ce89a9e2e4e67a6e8c7aa1559ffee207762cc328e358634664aa207aba"}
Nov 01 12:38:54.199261 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:54.199238    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp" event={"ID":"b5d15fcb-133f-40dd-b873-929f1b71a460","Type":"ContainerStarted","Data":"12f2b263de4ce45cdd550c4f1c27ccf0c3e30eb78ce958835694fa7d317192a7"}
Nov 01 12:38:55.207625 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.207545    5725 generic.go:356] "Generic (PLEG): container finished" podID="b5d15fcb-133f-40dd-b873-929f1b71a460" containerID="c9039916d341d8c0d48a5292a3cac2fb91a5bc11ffb5dbc301eee0509b7f6c29" exitCode=0
Nov 01 12:38:55.208022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.207627    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp" event={"ID":"b5d15fcb-133f-40dd-b873-929f1b71a460","Type":"ContainerDied","Data":"c9039916d341d8c0d48a5292a3cac2fb91a5bc11ffb5dbc301eee0509b7f6c29"}
Nov 01 12:38:55.405687 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.405658    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:55.476427 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.476399    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-xks-ca\") pod \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") "
Nov 01 12:38:55.476592 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.476535    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-openshift-ca\") pod \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") "
Nov 01 12:38:55.476592 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.476570    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-kube-api-access-gsb95\" (UniqueName: \"kubernetes.io/projected/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-kube-api-access-gsb95\") pod \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\" (UID: \"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96\") "
Nov 01 12:38:55.476882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.476857    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-openshift-ca" pod "8cc355f7-12e8-42a5-bf79-1b9ef6be7d96" (UID: "8cc355f7-12e8-42a5-bf79-1b9ef6be7d96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:55.478627 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.478599    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-xks-ca" pod "8cc355f7-12e8-42a5-bf79-1b9ef6be7d96" (UID: "8cc355f7-12e8-42a5-bf79-1b9ef6be7d96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:38:55.478736 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.478717    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-kube-api-access-gsb95" pod "8cc355f7-12e8-42a5-bf79-1b9ef6be7d96" (UID: "8cc355f7-12e8-42a5-bf79-1b9ef6be7d96"). InnerVolumeSpecName "kube-api-access-gsb95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:55.578129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.578083    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:55.578129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.578116    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:55.578129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:55.578126    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsb95\" (UniqueName: \"kubernetes.io/projected/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96-kube-api-access-gsb95\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:56.215619 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.215590    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"
Nov 01 12:38:56.216032 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.215630    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx" event={"ID":"8cc355f7-12e8-42a5-bf79-1b9ef6be7d96","Type":"ContainerDied","Data":"bc423ca115b474649374d979baab793270f9a30821f60a2e8afb9322e5488b17"}
Nov 01 12:38:56.216032 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.215675    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc423ca115b474649374d979baab793270f9a30821f60a2e8afb9322e5488b17"
Nov 01 12:38:56.371724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.371698    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:56.488413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.488374    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d15fcb-133f-40dd-b873-929f1b71a460-kube-api-access-n9m4d\" (UniqueName: \"kubernetes.io/projected/b5d15fcb-133f-40dd-b873-929f1b71a460-kube-api-access-n9m4d\") pod \"b5d15fcb-133f-40dd-b873-929f1b71a460\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") "
Nov 01 12:38:56.488606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.488469    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-openshift-ca\") pod \"b5d15fcb-133f-40dd-b873-929f1b71a460\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") "
Nov 01 12:38:56.488681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.488628    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-xks-ca\") pod \"b5d15fcb-133f-40dd-b873-929f1b71a460\" (UID: \"b5d15fcb-133f-40dd-b873-929f1b71a460\") "
Nov 01 12:38:56.488870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.488848    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-openshift-ca" pod "b5d15fcb-133f-40dd-b873-929f1b71a460" (UID: "b5d15fcb-133f-40dd-b873-929f1b71a460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:38:56.488984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.488969    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:56.491831 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.491802    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-xks-ca" pod "b5d15fcb-133f-40dd-b873-929f1b71a460" (UID: "b5d15fcb-133f-40dd-b873-929f1b71a460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:38:56.491961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.491842    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d15fcb-133f-40dd-b873-929f1b71a460-kube-api-access-n9m4d" pod "b5d15fcb-133f-40dd-b873-929f1b71a460" (UID: "b5d15fcb-133f-40dd-b873-929f1b71a460"). InnerVolumeSpecName "kube-api-access-n9m4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:38:56.589745 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.589713    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9m4d\" (UniqueName: \"kubernetes.io/projected/b5d15fcb-133f-40dd-b873-929f1b71a460-kube-api-access-n9m4d\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:56.589745 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:56.589743    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d15fcb-133f-40dd-b873-929f1b71a460-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:38:57.227813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:57.227778    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp" event={"ID":"b5d15fcb-133f-40dd-b873-929f1b71a460","Type":"ContainerDied","Data":"12f2b263de4ce45cdd550c4f1c27ccf0c3e30eb78ce958835694fa7d317192a7"}
Nov 01 12:38:57.228265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:57.227836    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12f2b263de4ce45cdd550c4f1c27ccf0c3e30eb78ce958835694fa7d317192a7"
Nov 01 12:38:57.228265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:57.227791    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"
Nov 01 12:38:59.218905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.218867    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"]
Nov 01 12:38:59.219399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.219385    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8cc355f7-12e8-42a5-bf79-1b9ef6be7d96" containerName="revoke-keys"
Nov 01 12:38:59.219538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.219399    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc355f7-12e8-42a5-bf79-1b9ef6be7d96" containerName="revoke-keys"
Nov 01 12:38:59.219538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.219414    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b5d15fcb-133f-40dd-b873-929f1b71a460" containerName="revoke-keys"
Nov 01 12:38:59.219538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.219419    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d15fcb-133f-40dd-b873-929f1b71a460" containerName="revoke-keys"
Nov 01 12:38:59.219538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.219491    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc355f7-12e8-42a5-bf79-1b9ef6be7d96" containerName="revoke-keys"
Nov 01 12:38:59.219538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.219500    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d15fcb-133f-40dd-b873-929f1b71a460" containerName="revoke-keys"
Nov 01 12:38:59.223106 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.223085    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.324495 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.324452    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.324683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.324513    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.324683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.324566    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f8581-c11d-4354-b525-297dc8159f8a-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.324683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.324610    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.324683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.324676    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.324892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.324704    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f8581-c11d-4354-b525-297dc8159f8a-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.324892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.324733    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-kube-api-access-dpxq4\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.324892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.324772    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.324892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.324800    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.328203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.328169    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"]
Nov 01 12:38:59.425853 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.425821    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.425853 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.425857    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f8581-c11d-4354-b525-297dc8159f8a-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426143 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.425881    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-kube-api-access-dpxq4\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426143 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.425918    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426143 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.425944    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426143 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.426021    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426143 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.426087    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426143 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.426141    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f8581-c11d-4354-b525-297dc8159f8a-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426446 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.426187    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426446 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.426363    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426446 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.426383    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426541 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.426460    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.426617    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.426816 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.426795    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f8581-c11d-4354-b525-297dc8159f8a-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.428453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.428428    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f8581-c11d-4354-b525-297dc8159f8a-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.428875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.428853    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.479748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.479720    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.480689 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.480664    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-kube-api-access-dpxq4\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.538299 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.538264    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:38:59.981314 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.981281    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"]
Nov 01 12:38:59.988448 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:38:59.988415    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285f8581_c11d_4354_b525_297dc8159f8a.slice/crio-0aa17070bbebde2a8cbf5490e0dabfaf0f0ebe07083a4db4bd18d2295e9542c6 WatchSource:0}: Error finding container 0aa17070bbebde2a8cbf5490e0dabfaf0f0ebe07083a4db4bd18d2295e9542c6: Status 404 returned error can't find the container with id 0aa17070bbebde2a8cbf5490e0dabfaf0f0ebe07083a4db4bd18d2295e9542c6
Nov 01 12:38:59.992786 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.992753    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:38:59.992883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.992823    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:38:59.992883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:38:59.992854    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:00.253243 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:00.253152    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" event={"ID":"285f8581-c11d-4354-b525-297dc8159f8a","Type":"ContainerStarted","Data":"fd5d586fc512abb6cfbdad01ee753795787abac1c316b059fb3bbfcaaccec75c"}
Nov 01 12:39:00.253637 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:00.253254    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" event={"ID":"285f8581-c11d-4354-b525-297dc8159f8a","Type":"ContainerStarted","Data":"0aa17070bbebde2a8cbf5490e0dabfaf0f0ebe07083a4db4bd18d2295e9542c6"}
Nov 01 12:39:00.363736 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:00.363674    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podStartSLOduration=1.363659274 podStartE2EDuration="1.363659274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:00.361607907 +0000 UTC m=+981.868799125" watchObservedRunningTime="2026-11-01 12:39:00.363659274 +0000 UTC m=+981.870850492"
Nov 01 12:39:00.539342 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:00.539264    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:39:01.059075 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:39:01.049333    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:39:01.559661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:01.558188    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 12:39:01.559661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:01.558256    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 12:39:02.541075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:02.540449    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 12:39:02.541075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:02.540511    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 12:39:03.545077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.542570    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"]
Nov 01 12:39:03.558077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.557619    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 12:39:03.558077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.557684    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 12:39:03.561913 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.561887    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.586843 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.586808    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"]
Nov 01 12:39:03.635293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.635260    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:39:03.636275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.636251    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:39:03.689417 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.689375    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b66d97e-lnrfx\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.689571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.689519    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b66d97e-lnrfx\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.689694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.689668    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/ad9115e0-22b8-443f-a831-5953676a38d3-kube-api-access-wdpss\") pod \"maas-api-revoke-sub-5b66d97e-lnrfx\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.790643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.790600    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b66d97e-lnrfx\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.790814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.790685    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/ad9115e0-22b8-443f-a831-5953676a38d3-kube-api-access-wdpss\") pod \"maas-api-revoke-sub-5b66d97e-lnrfx\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.790814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.790714    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b66d97e-lnrfx\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.791322 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.791300    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b66d97e-lnrfx\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.793246 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.793219    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b66d97e-lnrfx\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.852998 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.852866    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/ad9115e0-22b8-443f-a831-5953676a38d3-kube-api-access-wdpss\") pod \"maas-api-revoke-sub-5b66d97e-lnrfx\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:03.877339 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:03.877299    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:04.297401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:04.297366    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"]
Nov 01 12:39:04.304619 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:04.304589    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9115e0_22b8_443f_a831_5953676a38d3.slice/crio-1c319a71b8723314be59bfd63749aaf1fc6ddae8d71bada2a10617549489599a WatchSource:0}: Error finding container 1c319a71b8723314be59bfd63749aaf1fc6ddae8d71bada2a10617549489599a: Status 404 returned error can't find the container with id 1c319a71b8723314be59bfd63749aaf1fc6ddae8d71bada2a10617549489599a
Nov 01 12:39:04.545180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:04.543456    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 12:39:04.545180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:04.543516    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 12:39:05.297083 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:05.297031    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx" event={"ID":"ad9115e0-22b8-443f-a831-5953676a38d3","Type":"ContainerStarted","Data":"26422fc1b9d1eb108bd858643e21b586e35b132df8c29b62ba5f2dc554eb4096"}
Nov 01 12:39:05.297277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:05.297111    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx" event={"ID":"ad9115e0-22b8-443f-a831-5953676a38d3","Type":"ContainerStarted","Data":"1c319a71b8723314be59bfd63749aaf1fc6ddae8d71bada2a10617549489599a"}
Nov 01 12:39:05.406022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:05.405963    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx" podStartSLOduration=2.405943805 podStartE2EDuration="2.405943805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:05.405288927 +0000 UTC m=+986.912480145" watchObservedRunningTime="2026-11-01 12:39:05.405943805 +0000 UTC m=+986.913135023"
Nov 01 12:39:05.547077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:05.542436    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 12:39:05.547077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:05.542498    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 12:39:06.308107 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:06.308072    5725 generic.go:356] "Generic (PLEG): container finished" podID="ad9115e0-22b8-443f-a831-5953676a38d3" containerID="26422fc1b9d1eb108bd858643e21b586e35b132df8c29b62ba5f2dc554eb4096" exitCode=0
Nov 01 12:39:06.308307 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:06.308111    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx" event={"ID":"ad9115e0-22b8-443f-a831-5953676a38d3","Type":"ContainerDied","Data":"26422fc1b9d1eb108bd858643e21b586e35b132df8c29b62ba5f2dc554eb4096"}
Nov 01 12:39:06.539270 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:06.539228    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 01 12:39:06.539481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:06.539281    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 01 12:39:07.478591 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.478560    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:07.543445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.543414    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:39:07.543730 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.543708    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:39:07.544519 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.544498    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:39:07.634328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.634228    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9115e0-22b8-443f-a831-5953676a38d3-kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/ad9115e0-22b8-443f-a831-5953676a38d3-kube-api-access-wdpss\") pod \"ad9115e0-22b8-443f-a831-5953676a38d3\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") "
Nov 01 12:39:07.634511 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.634378    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-openshift-ca\") pod \"ad9115e0-22b8-443f-a831-5953676a38d3\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") "
Nov 01 12:39:07.634511 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.634431    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-xks-ca\") pod \"ad9115e0-22b8-443f-a831-5953676a38d3\" (UID: \"ad9115e0-22b8-443f-a831-5953676a38d3\") "
Nov 01 12:39:07.634755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.634729    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-openshift-ca" pod "ad9115e0-22b8-443f-a831-5953676a38d3" (UID: "ad9115e0-22b8-443f-a831-5953676a38d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:07.634949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.634926    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:07.636629 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.636602    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-xks-ca" pod "ad9115e0-22b8-443f-a831-5953676a38d3" (UID: "ad9115e0-22b8-443f-a831-5953676a38d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:07.636906 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.636880    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9115e0-22b8-443f-a831-5953676a38d3-kube-api-access-wdpss" pod "ad9115e0-22b8-443f-a831-5953676a38d3" (UID: "ad9115e0-22b8-443f-a831-5953676a38d3"). InnerVolumeSpecName "kube-api-access-wdpss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:07.737003 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.736965    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9115e0-22b8-443f-a831-5953676a38d3-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:07.737003 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:07.737004    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdpss\" (UniqueName: \"kubernetes.io/projected/ad9115e0-22b8-443f-a831-5953676a38d3-kube-api-access-wdpss\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:08.323606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:08.323572    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"
Nov 01 12:39:08.323606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:08.323592    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx" event={"ID":"ad9115e0-22b8-443f-a831-5953676a38d3","Type":"ContainerDied","Data":"1c319a71b8723314be59bfd63749aaf1fc6ddae8d71bada2a10617549489599a"}
Nov 01 12:39:08.323892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:08.323645    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c319a71b8723314be59bfd63749aaf1fc6ddae8d71bada2a10617549489599a"
Nov 01 12:39:11.940460 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:11.940419    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"]
Nov 01 12:39:11.940952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:11.940914    5725 cpu_manager.go:414] "containerMap: removing container" podUID="ad9115e0-22b8-443f-a831-5953676a38d3" containerName="revoke-keys"
Nov 01 12:39:11.940952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:11.940926    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9115e0-22b8-443f-a831-5953676a38d3" containerName="revoke-keys"
Nov 01 12:39:11.941098 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:11.941027    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9115e0-22b8-443f-a831-5953676a38d3" containerName="revoke-keys"
Nov 01 12:39:11.944626 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:11.944604    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:11.997602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:11.997568    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"]
Nov 01 12:39:12.040568 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.040539    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:39:12.040725 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.040622    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:39:12.084694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.084656    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e688374-wpjbk\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.084694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.084696    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e688374-wpjbk\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.084917 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.084721    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8hk\" (UniqueName: \"kubernetes.io/projected/11363d9f-3a67-4427-abaf-c32e25968810-kube-api-access-kf8hk\") pod \"maas-api-revoke-sub-1e688374-wpjbk\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.186246 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.186216    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e688374-wpjbk\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.186246 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.186252    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e688374-wpjbk\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.186512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.186277    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8hk\" (UniqueName: \"kubernetes.io/projected/11363d9f-3a67-4427-abaf-c32e25968810-kube-api-access-kf8hk\") pod \"maas-api-revoke-sub-1e688374-wpjbk\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.186894 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.186871    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e688374-wpjbk\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.188699 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.188678    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e688374-wpjbk\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.240081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.240030    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8hk\" (UniqueName: \"kubernetes.io/projected/11363d9f-3a67-4427-abaf-c32e25968810-kube-api-access-kf8hk\") pod \"maas-api-revoke-sub-1e688374-wpjbk\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.262123 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.262096    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:12.637362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:12.637272    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"]
Nov 01 12:39:12.645193 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:12.645160    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11363d9f_3a67_4427_abaf_c32e25968810.slice/crio-62160318bb8095630290732dc444156e95a9de9b951dbc031153a552828e9203 WatchSource:0}: Error finding container 62160318bb8095630290732dc444156e95a9de9b951dbc031153a552828e9203: Status 404 returned error can't find the container with id 62160318bb8095630290732dc444156e95a9de9b951dbc031153a552828e9203
Nov 01 12:39:13.324765 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.324727    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"]
Nov 01 12:39:13.328750 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.328729    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.363020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.362983    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk" event={"ID":"11363d9f-3a67-4427-abaf-c32e25968810","Type":"ContainerStarted","Data":"84ea4363d7b64ac36c38773bbbec36ed66869ff3bfedb3104f2d381c69abe8e1"}
Nov 01 12:39:13.363206 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.363080    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk" event={"ID":"11363d9f-3a67-4427-abaf-c32e25968810","Type":"ContainerStarted","Data":"62160318bb8095630290732dc444156e95a9de9b951dbc031153a552828e9203"}
Nov 01 12:39:13.381592 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.381555    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"]
Nov 01 12:39:13.503536 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.503503    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823a85ef-j6585\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.503755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.503546    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823a85ef-j6585\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.503836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.503785    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2tn\" (UniqueName: \"kubernetes.io/projected/616b7f3c-0d6e-4f3d-8df2-594665010978-kube-api-access-4v2tn\") pod \"maas-api-revoke-sub-823a85ef-j6585\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.541833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.541782    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk" podStartSLOduration=2.54176807 podStartE2EDuration="2.54176807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:13.541350514 +0000 UTC m=+995.048541728" watchObservedRunningTime="2026-11-01 12:39:13.54176807 +0000 UTC m=+995.048959293"
Nov 01 12:39:13.605134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.605100    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823a85ef-j6585\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.605272 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.605151    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823a85ef-j6585\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.605272 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.605245    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2tn\" (UniqueName: \"kubernetes.io/projected/616b7f3c-0d6e-4f3d-8df2-594665010978-kube-api-access-4v2tn\") pod \"maas-api-revoke-sub-823a85ef-j6585\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.605751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.605726    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823a85ef-j6585\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.607401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.607378    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823a85ef-j6585\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.659336 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.659303    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2tn\" (UniqueName: \"kubernetes.io/projected/616b7f3c-0d6e-4f3d-8df2-594665010978-kube-api-access-4v2tn\") pod \"maas-api-revoke-sub-823a85ef-j6585\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:13.943424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:13.943347    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:14.187141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.187104    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"]
Nov 01 12:39:14.192649 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.192625    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.234116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.234079    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"]
Nov 01 12:39:14.252379 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.252336    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"]
Nov 01 12:39:14.252545 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.252480    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.257937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.257910    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.283238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.283204    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"]
Nov 01 12:39:14.288389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.288349    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"]
Nov 01 12:39:14.328153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.328110    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqz4\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-kube-api-access-knqz4\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.328655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.328164    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.328655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.328289    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.328655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.328323    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.328655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.328346    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.328655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.328384    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.328655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.328419    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.328655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.328452    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.328655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.328479    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.329710 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.329192    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5b1bd4-openshift-default-dockercfg-4g7mc\""
Nov 01 12:39:14.338387 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.334316    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"]
Nov 01 12:39:14.338387 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.337753    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"]
Nov 01 12:39:14.376621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.376588    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585" event={"ID":"616b7f3c-0d6e-4f3d-8df2-594665010978","Type":"ContainerStarted","Data":"fd654c8fd8706cb6c487b83d196a74e89d1c882bb5c13413033b66b33924181d"}
Nov 01 12:39:14.378941 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.378913    5725 generic.go:356] "Generic (PLEG): container finished" podID="11363d9f-3a67-4427-abaf-c32e25968810" containerID="84ea4363d7b64ac36c38773bbbec36ed66869ff3bfedb3104f2d381c69abe8e1" exitCode=0
Nov 01 12:39:14.379114 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.378968    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk" event={"ID":"11363d9f-3a67-4427-abaf-c32e25968810","Type":"ContainerDied","Data":"84ea4363d7b64ac36c38773bbbec36ed66869ff3bfedb3104f2d381c69abe8e1"}
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.429777    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.429830    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.429901    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtv6s\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-kube-api-access-jtv6s\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.429930    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.429961    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64452fea-f72f-436f-b46d-ee9174d0f797-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430013    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430067    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430093    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430123    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430147    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djg5h\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-kube-api-access-djg5h\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430186    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430212    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430255    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430283    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430313    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64452fea-f72f-436f-b46d-ee9174d0f797-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430347    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430377    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430403    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430456    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430489    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430520    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430550    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430593    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69a426-38af-4e4a-a378-86100a1a8aac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430646    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqz4\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-kube-api-access-knqz4\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430676    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430705    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.430754    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.431731    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.431925    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.432015    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.432575    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.435105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.432858    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.439296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.439264    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.443896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.443865    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.486193 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.486156    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.486372 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.486314    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqz4\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-kube-api-access-knqz4\") pod \"maas-default-gateway-openshift-default-7df854bf67-vw7gq\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.513326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.513294    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:14.531999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.531965    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.532184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532012    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.532184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532085    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.532184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532109    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.532362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532183    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtv6s\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-kube-api-access-jtv6s\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.532362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532216    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.532362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532246    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64452fea-f72f-436f-b46d-ee9174d0f797-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.532362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532310    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.532362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532337    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djg5h\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-kube-api-access-djg5h\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532379    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532409    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532640    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532830    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532872    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64452fea-f72f-436f-b46d-ee9174d0f797-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532911    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532962    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.532993    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.533025    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.533027    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.533079    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.533123    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69a426-38af-4e4a-a378-86100a1a8aac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.533593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.533155    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.549413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.533958    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.549413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.533993    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64452fea-f72f-436f-b46d-ee9174d0f797-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.549413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.534282    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.549413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.535086    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.549413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.535289    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69a426-38af-4e4a-a378-86100a1a8aac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.549413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.535585    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.549413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.538259    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64452fea-f72f-436f-b46d-ee9174d0f797-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.549413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.541145    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.549413 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.541658    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.608075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.607385    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.617074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.615589    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtv6s\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-kube-api-access-jtv6s\") pod \"maas-default-gateway-openshift-default-7df854bf67-tpczd\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.622075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.617530    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:14.630476 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.629281    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.645288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.633592    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.645288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.634255    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djg5h\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-kube-api-access-djg5h\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:14.877549 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:14.877302    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:15.162133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:15.126724    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"]
Nov 01 12:39:15.261361 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:15.261325    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0b4044_3bc3_4ede_a6a1_e5d4973db18b.slice/crio-f7960dfb6e1e0921f4f34f5cc31e89ac105debfcf4b956565336af2738acf508 WatchSource:0}: Error finding container f7960dfb6e1e0921f4f34f5cc31e89ac105debfcf4b956565336af2738acf508: Status 404 returned error can't find the container with id f7960dfb6e1e0921f4f34f5cc31e89ac105debfcf4b956565336af2738acf508
Nov 01 12:39:15.311515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:15.311468    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:15.317246 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:15.317197    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:15.317520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:15.317493    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:15.526753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:15.526717    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" event={"ID":"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b","Type":"ContainerStarted","Data":"f7960dfb6e1e0921f4f34f5cc31e89ac105debfcf4b956565336af2738acf508"}
Nov 01 12:39:15.620074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:15.617184    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585" event={"ID":"616b7f3c-0d6e-4f3d-8df2-594665010978","Type":"ContainerStarted","Data":"c3543a8a58c7ee209d8a02a1965a4e0a645f3a04ba8dc130035eaf569ae5a69e"}
Nov 01 12:39:15.851197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:15.851112    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585" podStartSLOduration=2.851091471 podStartE2EDuration="2.851091471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:15.783851659 +0000 UTC m=+997.291042876" watchObservedRunningTime="2026-11-01 12:39:15.851091471 +0000 UTC m=+997.358282690"
Nov 01 12:39:16.016945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.016899    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"]
Nov 01 12:39:16.081284 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.079975    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:16.081284 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.080096    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:16.081284 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.080150    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:16.309277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.309225    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:39:16.512073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.510139    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:39:16.585196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.585142    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" event={"ID":"7b69a426-38af-4e4a-a378-86100a1a8aac","Type":"ContainerStarted","Data":"ace9539b13f24afc178373e5f658ea7a4e7061aff1146fc43cdc898b53c6c968"}
Nov 01 12:39:16.589887 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.589851    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" event={"ID":"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b","Type":"ContainerStarted","Data":"1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85"}
Nov 01 12:39:16.592448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.592414    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585" event={"ID":"616b7f3c-0d6e-4f3d-8df2-594665010978","Type":"ContainerDied","Data":"c3543a8a58c7ee209d8a02a1965a4e0a645f3a04ba8dc130035eaf569ae5a69e"}
Nov 01 12:39:16.592643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.592172    5725 generic.go:356] "Generic (PLEG): container finished" podID="616b7f3c-0d6e-4f3d-8df2-594665010978" containerID="c3543a8a58c7ee209d8a02a1965a4e0a645f3a04ba8dc130035eaf569ae5a69e" exitCode=0
Nov 01 12:39:16.714610 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.714413    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podStartSLOduration=2.714389027 podStartE2EDuration="2.714389027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:16.701511506 +0000 UTC m=+998.208702724" watchObservedRunningTime="2026-11-01 12:39:16.714389027 +0000 UTC m=+998.221580245"
Nov 01 12:39:16.873998 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.873968    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:16.926140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.926103    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11363d9f-3a67-4427-abaf-c32e25968810-kube-api-access-kf8hk\" (UniqueName: \"kubernetes.io/projected/11363d9f-3a67-4427-abaf-c32e25968810-kube-api-access-kf8hk\") pod \"11363d9f-3a67-4427-abaf-c32e25968810\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") "
Nov 01 12:39:16.926311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.926196    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-openshift-ca\") pod \"11363d9f-3a67-4427-abaf-c32e25968810\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") "
Nov 01 12:39:16.926358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.926343    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-xks-ca\") pod \"11363d9f-3a67-4427-abaf-c32e25968810\" (UID: \"11363d9f-3a67-4427-abaf-c32e25968810\") "
Nov 01 12:39:16.926663 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.926623    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-openshift-ca" pod "11363d9f-3a67-4427-abaf-c32e25968810" (UID: "11363d9f-3a67-4427-abaf-c32e25968810"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:16.926849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.926826    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:16.929094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.928654    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11363d9f-3a67-4427-abaf-c32e25968810-kube-api-access-kf8hk" pod "11363d9f-3a67-4427-abaf-c32e25968810" (UID: "11363d9f-3a67-4427-abaf-c32e25968810"). InnerVolumeSpecName "kube-api-access-kf8hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:16.929094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:16.928942    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-xks-ca" pod "11363d9f-3a67-4427-abaf-c32e25968810" (UID: "11363d9f-3a67-4427-abaf-c32e25968810"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:17.028533 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.028483    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11363d9f-3a67-4427-abaf-c32e25968810-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:17.028746 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.028728    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf8hk\" (UniqueName: \"kubernetes.io/projected/11363d9f-3a67-4427-abaf-c32e25968810-kube-api-access-kf8hk\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:17.039073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.038176    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"]
Nov 01 12:39:17.050762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.050719    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:17.050891 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.050813    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:17.050891 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.050855    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:39:17.514325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.514285    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:17.515533 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.515494    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-vw7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:39:17.515668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.515561    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 12:39:17.611622 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.611596    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"
Nov 01 12:39:17.612240 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.611674    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk" event={"ID":"11363d9f-3a67-4427-abaf-c32e25968810","Type":"ContainerDied","Data":"62160318bb8095630290732dc444156e95a9de9b951dbc031153a552828e9203"}
Nov 01 12:39:17.612423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.612408    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62160318bb8095630290732dc444156e95a9de9b951dbc031153a552828e9203"
Nov 01 12:39:17.613762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.613739    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" event={"ID":"64452fea-f72f-436f-b46d-ee9174d0f797","Type":"ContainerStarted","Data":"7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620"}
Nov 01 12:39:17.613916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.613897    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" event={"ID":"64452fea-f72f-436f-b46d-ee9174d0f797","Type":"ContainerStarted","Data":"66dfaaf6b517a459adbd1aed4045206f65149f93cdb102a6cac705898fbccc9b"}
Nov 01 12:39:17.616978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.616950    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" event={"ID":"7b69a426-38af-4e4a-a378-86100a1a8aac","Type":"ContainerStarted","Data":"e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742"}
Nov 01 12:39:17.623348 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.623334    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:17.624486 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.624466    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tpczd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 12:39:17.624585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.624508    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 12:39:17.761075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.760415    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" podStartSLOduration=3.76039607 podStartE2EDuration="3.76039607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:17.757742098 +0000 UTC m=+999.264933316" watchObservedRunningTime="2026-11-01 12:39:17.76039607 +0000 UTC m=+999.267587287"
Nov 01 12:39:17.881083 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.880465    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:17.892933 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:17.892871    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podStartSLOduration=3.892857021 podStartE2EDuration="3.892857021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:17.889216717 +0000 UTC m=+999.396407946" watchObservedRunningTime="2026-11-01 12:39:17.892857021 +0000 UTC m=+999.400048298"
Nov 01 12:39:18.108185 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.107787    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:18.204447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.204404    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-xks-ca\") pod \"616b7f3c-0d6e-4f3d-8df2-594665010978\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") "
Nov 01 12:39:18.204759 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.204737    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-openshift-ca\") pod \"616b7f3c-0d6e-4f3d-8df2-594665010978\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") "
Nov 01 12:39:18.205010 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.204993    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616b7f3c-0d6e-4f3d-8df2-594665010978-kube-api-access-4v2tn\" (UniqueName: \"kubernetes.io/projected/616b7f3c-0d6e-4f3d-8df2-594665010978-kube-api-access-4v2tn\") pod \"616b7f3c-0d6e-4f3d-8df2-594665010978\" (UID: \"616b7f3c-0d6e-4f3d-8df2-594665010978\") "
Nov 01 12:39:18.206250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.206222    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-openshift-ca" pod "616b7f3c-0d6e-4f3d-8df2-594665010978" (UID: "616b7f3c-0d6e-4f3d-8df2-594665010978"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:18.208458 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.208431    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-xks-ca" pod "616b7f3c-0d6e-4f3d-8df2-594665010978" (UID: "616b7f3c-0d6e-4f3d-8df2-594665010978"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:18.212885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.212190    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616b7f3c-0d6e-4f3d-8df2-594665010978-kube-api-access-4v2tn" pod "616b7f3c-0d6e-4f3d-8df2-594665010978" (UID: "616b7f3c-0d6e-4f3d-8df2-594665010978"). InnerVolumeSpecName "kube-api-access-4v2tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:18.310091 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.306837    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:18.310091 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.306882    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/616b7f3c-0d6e-4f3d-8df2-594665010978-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:18.310091 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.306899    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v2tn\" (UniqueName: \"kubernetes.io/projected/616b7f3c-0d6e-4f3d-8df2-594665010978-kube-api-access-4v2tn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:18.514230 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.514196    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-vw7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:39:18.514501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.514471    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 12:39:18.624432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.624382    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tpczd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 12:39:18.625021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.624990    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 12:39:18.688074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.685837    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"
Nov 01 12:39:18.701079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.699182    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585" event={"ID":"616b7f3c-0d6e-4f3d-8df2-594665010978","Type":"ContainerDied","Data":"fd654c8fd8706cb6c487b83d196a74e89d1c882bb5c13413033b66b33924181d"}
Nov 01 12:39:18.701079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.699340    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd654c8fd8706cb6c487b83d196a74e89d1c882bb5c13413033b66b33924181d"
Nov 01 12:39:18.894178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.891312    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 12:39:18.894178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:18.891369    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 12:39:19.513943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.513913    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-vw7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:39:19.514199 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.514149    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 12:39:19.623292 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.623255    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tpczd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 12:39:19.623458 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.623306    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 12:39:19.807135 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.786531    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"]
Nov 01 12:39:19.808631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.808596    5725 cpu_manager.go:414] "containerMap: removing container" podUID="616b7f3c-0d6e-4f3d-8df2-594665010978" containerName="revoke-keys"
Nov 01 12:39:19.813093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.812732    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616b7f3c-0d6e-4f3d-8df2-594665010978" containerName="revoke-keys"
Nov 01 12:39:19.813093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.812781    5725 cpu_manager.go:414] "containerMap: removing container" podUID="11363d9f-3a67-4427-abaf-c32e25968810" containerName="revoke-keys"
Nov 01 12:39:19.813093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.812790    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11363d9f-3a67-4427-abaf-c32e25968810" containerName="revoke-keys"
Nov 01 12:39:19.813093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.812957    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="11363d9f-3a67-4427-abaf-c32e25968810" containerName="revoke-keys"
Nov 01 12:39:19.813093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.812981    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="616b7f3c-0d6e-4f3d-8df2-594665010978" containerName="revoke-keys"
Nov 01 12:39:19.817909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.817365    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:19.821127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.820385    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"]
Nov 01 12:39:19.946129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.923958    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 12:39:19.946129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.924011    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 12:39:19.946129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.926182    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:39:19.985086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.955402    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:39:19.985086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.956103    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntm2x\" (UniqueName: \"kubernetes.io/projected/3d68db58-a145-4f99-98d5-9a3df9919c4f-kube-api-access-ntm2x\") pod \"maas-api-revoke-sub-0468bc6e-2kdt8\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:19.985086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.956147    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0468bc6e-2kdt8\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:19.985086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.956345    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0468bc6e-2kdt8\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:19.985086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:19.985021    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"]
Nov 01 12:39:20.024942 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.021124    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"]
Nov 01 12:39:20.024942 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.023873    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.118350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.114840    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d88c85-7drzz\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.118350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.114909    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0468bc6e-2kdt8\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:20.118350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.115018    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntm2x\" (UniqueName: \"kubernetes.io/projected/3d68db58-a145-4f99-98d5-9a3df9919c4f-kube-api-access-ntm2x\") pod \"maas-api-revoke-sub-0468bc6e-2kdt8\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:20.118350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.115073    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0468bc6e-2kdt8\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:20.118350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.115118    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spxm8\" (UniqueName: \"kubernetes.io/projected/985b82b0-c281-4f06-9b8c-12956a950dd9-kube-api-access-spxm8\") pod \"maas-api-revoke-sub-36d88c85-7drzz\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.118350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.115183    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d88c85-7drzz\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.118350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.116249    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0468bc6e-2kdt8\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:20.118350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.118078    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0468bc6e-2kdt8\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:20.272089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.263508    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d88c85-7drzz\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.272089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.263675    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d88c85-7drzz\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.272089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.263804    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spxm8\" (UniqueName: \"kubernetes.io/projected/985b82b0-c281-4f06-9b8c-12956a950dd9-kube-api-access-spxm8\") pod \"maas-api-revoke-sub-36d88c85-7drzz\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.315969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.315910    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d88c85-7drzz\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.317767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.317714    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntm2x\" (UniqueName: \"kubernetes.io/projected/3d68db58-a145-4f99-98d5-9a3df9919c4f-kube-api-access-ntm2x\") pod \"maas-api-revoke-sub-0468bc6e-2kdt8\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:20.411133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.319784    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spxm8\" (UniqueName: \"kubernetes.io/projected/985b82b0-c281-4f06-9b8c-12956a950dd9-kube-api-access-spxm8\") pod \"maas-api-revoke-sub-36d88c85-7drzz\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.416554 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.416520    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d88c85-7drzz\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:20.514570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.514529    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-vw7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:39:20.514738 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.514613    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 12:39:20.615148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.615115    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:20.624157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.624116    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tpczd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 12:39:20.624293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.624203    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 12:39:20.677799 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:20.677772    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:21.067081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.065077    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 12:39:21.067081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.065134    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 12:39:21.423407 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:21.421728    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d68db58_a145_4f99_98d5_9a3df9919c4f.slice/crio-8813403ce3e53ec3302ab4ba3b3f11c4235f0297bf114b93c93e2a1e5f28a68a WatchSource:0}: Error finding container 8813403ce3e53ec3302ab4ba3b3f11c4235f0297bf114b93c93e2a1e5f28a68a: Status 404 returned error can't find the container with id 8813403ce3e53ec3302ab4ba3b3f11c4235f0297bf114b93c93e2a1e5f28a68a
Nov 01 12:39:21.423407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.422994    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"]
Nov 01 12:39:21.515217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.514756    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-vw7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:39:21.515217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.514830    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 12:39:21.624025 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.623991    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tpczd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 12:39:21.624211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.624062    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 12:39:21.689762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.681729    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"]
Nov 01 12:39:21.716709 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:21.716675    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985b82b0_c281_4f06_9b8c_12956a950dd9.slice/crio-f4cbbb6ac3033eb64558c70fc5828ccf9b56fda55022b071e1fd72c5e4374648 WatchSource:0}: Error finding container f4cbbb6ac3033eb64558c70fc5828ccf9b56fda55022b071e1fd72c5e4374648: Status 404 returned error can't find the container with id f4cbbb6ac3033eb64558c70fc5828ccf9b56fda55022b071e1fd72c5e4374648
Nov 01 12:39:21.823300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.823261    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8" event={"ID":"3d68db58-a145-4f99-98d5-9a3df9919c4f","Type":"ContainerStarted","Data":"8813403ce3e53ec3302ab4ba3b3f11c4235f0297bf114b93c93e2a1e5f28a68a"}
Nov 01 12:39:21.825134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.825096    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz" event={"ID":"985b82b0-c281-4f06-9b8c-12956a950dd9","Type":"ContainerStarted","Data":"f4cbbb6ac3033eb64558c70fc5828ccf9b56fda55022b071e1fd72c5e4374648"}
Nov 01 12:39:21.891072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.889230    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 12:39:21.891072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:21.889292    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 12:39:22.516480 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:22.516441    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-vw7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:39:22.517076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:22.517012    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 12:39:22.629369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:22.629162    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tpczd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 12:39:22.629369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:22.629238    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 12:39:22.863402 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:22.862500    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8" event={"ID":"3d68db58-a145-4f99-98d5-9a3df9919c4f","Type":"ContainerStarted","Data":"8431628f2549606e94150bec33f4f0bd0f5197bfa4c866b2317f1186ef028251"}
Nov 01 12:39:22.865777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:22.865741    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz" event={"ID":"985b82b0-c281-4f06-9b8c-12956a950dd9","Type":"ContainerStarted","Data":"b0a296e6c2e761a495c13f07b78cc6d9f6af521314e273b11efc823c3fd3952a"}
Nov 01 12:39:22.887534 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:22.887443    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 12:39:22.887794 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:22.887753    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 12:39:23.002644 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.002564    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8" podStartSLOduration=4.002545215 podStartE2EDuration="4.002545215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:23.00065271 +0000 UTC m=+1004.507843954" watchObservedRunningTime="2026-11-01 12:39:23.002545215 +0000 UTC m=+1004.509736434"
Nov 01 12:39:23.072075 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:39:23.065799    5725 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d68db58_a145_4f99_98d5_9a3df9919c4f.slice/crio-conmon-8431628f2549606e94150bec33f4f0bd0f5197bfa4c866b2317f1186ef028251.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d68db58_a145_4f99_98d5_9a3df9919c4f.slice/crio-8431628f2549606e94150bec33f4f0bd0f5197bfa4c866b2317f1186ef028251.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 12:39:23.132072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.129339    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz" podStartSLOduration=4.129318639 podStartE2EDuration="4.129318639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:23.125288588 +0000 UTC m=+1004.632479815" watchObservedRunningTime="2026-11-01 12:39:23.129318639 +0000 UTC m=+1004.636509857"
Nov 01 12:39:23.522169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.521692    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-vw7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:39:23.522169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.521753    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 12:39:23.640071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.629145    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tpczd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 12:39:23.640071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.629209    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 12:39:23.886219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.885428    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 12:39:23.886219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.885486    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 12:39:23.890997 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.890965    5725 generic.go:356] "Generic (PLEG): container finished" podID="985b82b0-c281-4f06-9b8c-12956a950dd9" containerID="b0a296e6c2e761a495c13f07b78cc6d9f6af521314e273b11efc823c3fd3952a" exitCode=0
Nov 01 12:39:23.891400 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.891376    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz" event={"ID":"985b82b0-c281-4f06-9b8c-12956a950dd9","Type":"ContainerDied","Data":"b0a296e6c2e761a495c13f07b78cc6d9f6af521314e273b11efc823c3fd3952a"}
Nov 01 12:39:23.897476 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.897448    5725 generic.go:356] "Generic (PLEG): container finished" podID="3d68db58-a145-4f99-98d5-9a3df9919c4f" containerID="8431628f2549606e94150bec33f4f0bd0f5197bfa4c866b2317f1186ef028251" exitCode=0
Nov 01 12:39:23.897724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:23.897706    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8" event={"ID":"3d68db58-a145-4f99-98d5-9a3df9919c4f","Type":"ContainerDied","Data":"8431628f2549606e94150bec33f4f0bd0f5197bfa4c866b2317f1186ef028251"}
Nov 01 12:39:24.515074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:24.514753    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-vw7gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:39:24.515074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:24.514820    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 01 12:39:24.623796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:24.623652    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-tpczd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 12:39:24.623796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:24.623709    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 12:39:24.907192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:24.905841    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:24.907192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:24.907163    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:24.908563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:24.908543    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:39:25.213252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.212865    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:25.218432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.218397    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:25.303450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.303392    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-xks-ca\") pod \"985b82b0-c281-4f06-9b8c-12956a950dd9\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") "
Nov 01 12:39:25.303640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.303576    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985b82b0-c281-4f06-9b8c-12956a950dd9-kube-api-access-spxm8\" (UniqueName: \"kubernetes.io/projected/985b82b0-c281-4f06-9b8c-12956a950dd9-kube-api-access-spxm8\") pod \"985b82b0-c281-4f06-9b8c-12956a950dd9\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") "
Nov 01 12:39:25.303704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.303636    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d68db58-a145-4f99-98d5-9a3df9919c4f-kube-api-access-ntm2x\" (UniqueName: \"kubernetes.io/projected/3d68db58-a145-4f99-98d5-9a3df9919c4f-kube-api-access-ntm2x\") pod \"3d68db58-a145-4f99-98d5-9a3df9919c4f\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") "
Nov 01 12:39:25.303792 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.303763    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-xks-ca\") pod \"3d68db58-a145-4f99-98d5-9a3df9919c4f\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") "
Nov 01 12:39:25.303847 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.303813    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-openshift-ca\") pod \"3d68db58-a145-4f99-98d5-9a3df9919c4f\" (UID: \"3d68db58-a145-4f99-98d5-9a3df9919c4f\") "
Nov 01 12:39:25.303909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.303847    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-openshift-ca\") pod \"985b82b0-c281-4f06-9b8c-12956a950dd9\" (UID: \"985b82b0-c281-4f06-9b8c-12956a950dd9\") "
Nov 01 12:39:25.304595 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.304474    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-openshift-ca" pod "985b82b0-c281-4f06-9b8c-12956a950dd9" (UID: "985b82b0-c281-4f06-9b8c-12956a950dd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:25.304736 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.304712    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-openshift-ca" pod "3d68db58-a145-4f99-98d5-9a3df9919c4f" (UID: "3d68db58-a145-4f99-98d5-9a3df9919c4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:25.305291 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.305255    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:25.305481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.305466    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:25.308445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.308077    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d68db58-a145-4f99-98d5-9a3df9919c4f-kube-api-access-ntm2x" pod "3d68db58-a145-4f99-98d5-9a3df9919c4f" (UID: "3d68db58-a145-4f99-98d5-9a3df9919c4f"). InnerVolumeSpecName "kube-api-access-ntm2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:25.308950 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.308921    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-xks-ca" pod "3d68db58-a145-4f99-98d5-9a3df9919c4f" (UID: "3d68db58-a145-4f99-98d5-9a3df9919c4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:25.310991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.310961    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-xks-ca" pod "985b82b0-c281-4f06-9b8c-12956a950dd9" (UID: "985b82b0-c281-4f06-9b8c-12956a950dd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:25.312494 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.312458    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985b82b0-c281-4f06-9b8c-12956a950dd9-kube-api-access-spxm8" pod "985b82b0-c281-4f06-9b8c-12956a950dd9" (UID: "985b82b0-c281-4f06-9b8c-12956a950dd9"). InnerVolumeSpecName "kube-api-access-spxm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:25.408073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.407295    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spxm8\" (UniqueName: \"kubernetes.io/projected/985b82b0-c281-4f06-9b8c-12956a950dd9-kube-api-access-spxm8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:25.408073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.407349    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntm2x\" (UniqueName: \"kubernetes.io/projected/3d68db58-a145-4f99-98d5-9a3df9919c4f-kube-api-access-ntm2x\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:25.408073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.407368    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d68db58-a145-4f99-98d5-9a3df9919c4f-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:25.408073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.407398    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985b82b0-c281-4f06-9b8c-12956a950dd9-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:25.923580 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.923546    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"
Nov 01 12:39:25.924778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.924258    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz" event={"ID":"985b82b0-c281-4f06-9b8c-12956a950dd9","Type":"ContainerDied","Data":"f4cbbb6ac3033eb64558c70fc5828ccf9b56fda55022b071e1fd72c5e4374648"}
Nov 01 12:39:25.924909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.924816    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4cbbb6ac3033eb64558c70fc5828ccf9b56fda55022b071e1fd72c5e4374648"
Nov 01 12:39:25.932266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.926178    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"
Nov 01 12:39:25.932266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.926255    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8" event={"ID":"3d68db58-a145-4f99-98d5-9a3df9919c4f","Type":"ContainerDied","Data":"8813403ce3e53ec3302ab4ba3b3f11c4235f0297bf114b93c93e2a1e5f28a68a"}
Nov 01 12:39:25.932266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:25.926318    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8813403ce3e53ec3302ab4ba3b3f11c4235f0297bf114b93c93e2a1e5f28a68a"
Nov 01 12:39:26.329298 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:26.329255    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:26.330239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:26.330215    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:26.330779 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:26.330759    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:39:26.402074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:26.401984    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:26.402451 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:26.402417    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:26.403904 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:26.403876    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:39:29.775360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.775328    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"]
Nov 01 12:39:29.775870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.775856    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3d68db58-a145-4f99-98d5-9a3df9919c4f" containerName="revoke-keys"
Nov 01 12:39:29.775916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.775871    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d68db58-a145-4f99-98d5-9a3df9919c4f" containerName="revoke-keys"
Nov 01 12:39:29.775916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.775890    5725 cpu_manager.go:414] "containerMap: removing container" podUID="985b82b0-c281-4f06-9b8c-12956a950dd9" containerName="revoke-keys"
Nov 01 12:39:29.775916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.775895    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985b82b0-c281-4f06-9b8c-12956a950dd9" containerName="revoke-keys"
Nov 01 12:39:29.776013 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.775973    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d68db58-a145-4f99-98d5-9a3df9919c4f" containerName="revoke-keys"
Nov 01 12:39:29.776013 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.775985    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="985b82b0-c281-4f06-9b8c-12956a950dd9" containerName="revoke-keys"
Nov 01 12:39:29.780610 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.780592    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:29.832593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.832557    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"]
Nov 01 12:39:29.870064 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.870013    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfgfs\" (UniqueName: \"kubernetes.io/projected/7e9ec72c-dcb2-4e39-8d8e-014287745a36-kube-api-access-sfgfs\") pod \"maas-api-revoke-sub-74bee244-bzhdq\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:29.870254 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.870096    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74bee244-bzhdq\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:29.870254 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.870206    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74bee244-bzhdq\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:29.873960 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.873935    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:39:29.874096 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.873989    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:39:29.972079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.972029    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfgfs\" (UniqueName: \"kubernetes.io/projected/7e9ec72c-dcb2-4e39-8d8e-014287745a36-kube-api-access-sfgfs\") pod \"maas-api-revoke-sub-74bee244-bzhdq\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:29.972268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.972128    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74bee244-bzhdq\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:29.972268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.972199    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74bee244-bzhdq\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:29.972815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.972788    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74bee244-bzhdq\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:29.974683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:29.974655    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74bee244-bzhdq\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:30.025767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.025693    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfgfs\" (UniqueName: \"kubernetes.io/projected/7e9ec72c-dcb2-4e39-8d8e-014287745a36-kube-api-access-sfgfs\") pod \"maas-api-revoke-sub-74bee244-bzhdq\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:30.056167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.056138    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"]
Nov 01 12:39:30.060401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.060380    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.094021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.093997    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:30.112612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.112562    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"]
Nov 01 12:39:30.176022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.175987    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdlwh\" (UniqueName: \"kubernetes.io/projected/437039d8-3fa5-4b26-b207-145ca139a447-kube-api-access-mdlwh\") pod \"maas-api-revoke-sub-124629a1-kx82v\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.176208 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.176096    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-124629a1-kx82v\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.176326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.176278    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-124629a1-kx82v\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.277619 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.277532    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdlwh\" (UniqueName: \"kubernetes.io/projected/437039d8-3fa5-4b26-b207-145ca139a447-kube-api-access-mdlwh\") pod \"maas-api-revoke-sub-124629a1-kx82v\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.277619 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.277599    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-124629a1-kx82v\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.277849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.277699    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-124629a1-kx82v\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.278363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.278344    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-124629a1-kx82v\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.279926 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.279908    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-124629a1-kx82v\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.331124 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.331086    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdlwh\" (UniqueName: \"kubernetes.io/projected/437039d8-3fa5-4b26-b207-145ca139a447-kube-api-access-mdlwh\") pod \"maas-api-revoke-sub-124629a1-kx82v\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.374766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.374734    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:30.476696 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.476660    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"]
Nov 01 12:39:30.481476 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:30.481442    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9ec72c_dcb2_4e39_8d8e_014287745a36.slice/crio-eed32ff863dd4870e3740d5321b1a4b3faa8299c8cdb089248659f8662552483 WatchSource:0}: Error finding container eed32ff863dd4870e3740d5321b1a4b3faa8299c8cdb089248659f8662552483: Status 404 returned error can't find the container with id eed32ff863dd4870e3740d5321b1a4b3faa8299c8cdb089248659f8662552483
Nov 01 12:39:30.618287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.618214    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"]
Nov 01 12:39:30.623563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.623543    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.665719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.665686    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"]
Nov 01 12:39:30.682311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.682274    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c99dbc5-d7hlk\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.682521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.682495    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdm7\" (UniqueName: \"kubernetes.io/projected/179f214c-6ed2-40dd-bda9-51841f2125a9-kube-api-access-tcdm7\") pod \"maas-api-revoke-sub-0c99dbc5-d7hlk\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.682613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.682550    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c99dbc5-d7hlk\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.749655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.749622    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"]
Nov 01 12:39:30.756294 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:30.756267    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437039d8_3fa5_4b26_b207_145ca139a447.slice/crio-909ac2181e3b069044089e09ef81dcc03c6c2f81cc3cc4741086129134a96203 WatchSource:0}: Error finding container 909ac2181e3b069044089e09ef81dcc03c6c2f81cc3cc4741086129134a96203: Status 404 returned error can't find the container with id 909ac2181e3b069044089e09ef81dcc03c6c2f81cc3cc4741086129134a96203
Nov 01 12:39:30.783811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.783779    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdm7\" (UniqueName: \"kubernetes.io/projected/179f214c-6ed2-40dd-bda9-51841f2125a9-kube-api-access-tcdm7\") pod \"maas-api-revoke-sub-0c99dbc5-d7hlk\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.784197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.783836    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c99dbc5-d7hlk\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.784197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.783936    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c99dbc5-d7hlk\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.784674 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.784652    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c99dbc5-d7hlk\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.786218 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.786194    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c99dbc5-d7hlk\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.837439 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.837417    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdm7\" (UniqueName: \"kubernetes.io/projected/179f214c-6ed2-40dd-bda9-51841f2125a9-kube-api-access-tcdm7\") pod \"maas-api-revoke-sub-0c99dbc5-d7hlk\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.951012 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.950982    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:30.979673 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.979637    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq" event={"ID":"7e9ec72c-dcb2-4e39-8d8e-014287745a36","Type":"ContainerStarted","Data":"ef8c458f8ea9635136e4712b23adbdd5f85374a764fa2f51b06c16bc816aa68d"}
Nov 01 12:39:30.979831 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.979725    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq" event={"ID":"7e9ec72c-dcb2-4e39-8d8e-014287745a36","Type":"ContainerStarted","Data":"eed32ff863dd4870e3740d5321b1a4b3faa8299c8cdb089248659f8662552483"}
Nov 01 12:39:30.983190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:30.983145    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v" event={"ID":"437039d8-3fa5-4b26-b207-145ca139a447","Type":"ContainerStarted","Data":"909ac2181e3b069044089e09ef81dcc03c6c2f81cc3cc4741086129134a96203"}
Nov 01 12:39:31.050013 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:39:31.049820    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:39:31.354686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:31.354634    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq" podStartSLOduration=2.354619644 podStartE2EDuration="2.354619644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:31.097180703 +0000 UTC m=+1012.604371921" watchObservedRunningTime="2026-11-01 12:39:31.354619644 +0000 UTC m=+1012.861810860"
Nov 01 12:39:31.356297 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:31.356271    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"]
Nov 01 12:39:31.363201 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:31.363168    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179f214c_6ed2_40dd_bda9_51841f2125a9.slice/crio-2c5a4c5a81ddbab3e1c1cbd636b3d6a2fbdb7c7c987e3f954fef875f00c1804a WatchSource:0}: Error finding container 2c5a4c5a81ddbab3e1c1cbd636b3d6a2fbdb7c7c987e3f954fef875f00c1804a: Status 404 returned error can't find the container with id 2c5a4c5a81ddbab3e1c1cbd636b3d6a2fbdb7c7c987e3f954fef875f00c1804a
Nov 01 12:39:31.992294 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:31.992256    5725 generic.go:356] "Generic (PLEG): container finished" podID="7e9ec72c-dcb2-4e39-8d8e-014287745a36" containerID="ef8c458f8ea9635136e4712b23adbdd5f85374a764fa2f51b06c16bc816aa68d" exitCode=0
Nov 01 12:39:31.992774 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:31.992334    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq" event={"ID":"7e9ec72c-dcb2-4e39-8d8e-014287745a36","Type":"ContainerDied","Data":"ef8c458f8ea9635136e4712b23adbdd5f85374a764fa2f51b06c16bc816aa68d"}
Nov 01 12:39:31.994109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:31.994080    5725 generic.go:356] "Generic (PLEG): container finished" podID="437039d8-3fa5-4b26-b207-145ca139a447" containerID="0db09bfc911669882e58a83fe92c5b33d6c828e1544b25f6d7bd4d59ccd9dbad" exitCode=0
Nov 01 12:39:31.994234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:31.994155    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v" event={"ID":"437039d8-3fa5-4b26-b207-145ca139a447","Type":"ContainerDied","Data":"0db09bfc911669882e58a83fe92c5b33d6c828e1544b25f6d7bd4d59ccd9dbad"}
Nov 01 12:39:31.995774 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:31.995744    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk" event={"ID":"179f214c-6ed2-40dd-bda9-51841f2125a9","Type":"ContainerStarted","Data":"2b6b08508a06d8dd86709bcb83389dd5931be2c13b60d9337366325956adad21"}
Nov 01 12:39:31.995885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:31.995794    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk" event={"ID":"179f214c-6ed2-40dd-bda9-51841f2125a9","Type":"ContainerStarted","Data":"2c5a4c5a81ddbab3e1c1cbd636b3d6a2fbdb7c7c987e3f954fef875f00c1804a"}
Nov 01 12:39:32.341732 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:32.341622    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk" podStartSLOduration=2.341606771 podStartE2EDuration="2.341606771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:32.338281464 +0000 UTC m=+1013.845472681" watchObservedRunningTime="2026-11-01 12:39:32.341606771 +0000 UTC m=+1013.848797989"
Nov 01 12:39:33.004020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.003981    5725 generic.go:356] "Generic (PLEG): container finished" podID="179f214c-6ed2-40dd-bda9-51841f2125a9" containerID="2b6b08508a06d8dd86709bcb83389dd5931be2c13b60d9337366325956adad21" exitCode=0
Nov 01 12:39:33.004463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.004081    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk" event={"ID":"179f214c-6ed2-40dd-bda9-51841f2125a9","Type":"ContainerDied","Data":"2b6b08508a06d8dd86709bcb83389dd5931be2c13b60d9337366325956adad21"}
Nov 01 12:39:33.200211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.200185    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:33.234896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.234869    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.313035    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-openshift-ca\") pod \"437039d8-3fa5-4b26-b207-145ca139a447\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") "
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.313139    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-openshift-ca\") pod \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") "
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.313173    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-xks-ca\") pod \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") "
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.313210    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-xks-ca\") pod \"437039d8-3fa5-4b26-b207-145ca139a447\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") "
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.313429    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9ec72c-dcb2-4e39-8d8e-014287745a36-kube-api-access-sfgfs\" (UniqueName: \"kubernetes.io/projected/7e9ec72c-dcb2-4e39-8d8e-014287745a36-kube-api-access-sfgfs\") pod \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\" (UID: \"7e9ec72c-dcb2-4e39-8d8e-014287745a36\") "
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.313439    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-openshift-ca" pod "437039d8-3fa5-4b26-b207-145ca139a447" (UID: "437039d8-3fa5-4b26-b207-145ca139a447"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.313502    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437039d8-3fa5-4b26-b207-145ca139a447-kube-api-access-mdlwh\" (UniqueName: \"kubernetes.io/projected/437039d8-3fa5-4b26-b207-145ca139a447-kube-api-access-mdlwh\") pod \"437039d8-3fa5-4b26-b207-145ca139a447\" (UID: \"437039d8-3fa5-4b26-b207-145ca139a447\") "
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.313884    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.316481    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-xks-ca" pod "7e9ec72c-dcb2-4e39-8d8e-014287745a36" (UID: "7e9ec72c-dcb2-4e39-8d8e-014287745a36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.316576    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-openshift-ca" pod "7e9ec72c-dcb2-4e39-8d8e-014287745a36" (UID: "7e9ec72c-dcb2-4e39-8d8e-014287745a36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:33.317846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.316984    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437039d8-3fa5-4b26-b207-145ca139a447-kube-api-access-mdlwh" pod "437039d8-3fa5-4b26-b207-145ca139a447" (UID: "437039d8-3fa5-4b26-b207-145ca139a447"). InnerVolumeSpecName "kube-api-access-mdlwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:33.318331 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.318261    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-xks-ca" pod "437039d8-3fa5-4b26-b207-145ca139a447" (UID: "437039d8-3fa5-4b26-b207-145ca139a447"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:33.318689 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.318666    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9ec72c-dcb2-4e39-8d8e-014287745a36-kube-api-access-sfgfs" pod "7e9ec72c-dcb2-4e39-8d8e-014287745a36" (UID: "7e9ec72c-dcb2-4e39-8d8e-014287745a36"). InnerVolumeSpecName "kube-api-access-sfgfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:33.414811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.414771    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:33.414811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.414813    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9ec72c-dcb2-4e39-8d8e-014287745a36-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:33.415081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.414830    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/437039d8-3fa5-4b26-b207-145ca139a447-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:33.415081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.414844    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfgfs\" (UniqueName: \"kubernetes.io/projected/7e9ec72c-dcb2-4e39-8d8e-014287745a36-kube-api-access-sfgfs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:33.415081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:33.414858    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdlwh\" (UniqueName: \"kubernetes.io/projected/437039d8-3fa5-4b26-b207-145ca139a447-kube-api-access-mdlwh\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:34.013203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.013151    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq" event={"ID":"7e9ec72c-dcb2-4e39-8d8e-014287745a36","Type":"ContainerDied","Data":"eed32ff863dd4870e3740d5321b1a4b3faa8299c8cdb089248659f8662552483"}
Nov 01 12:39:34.013641 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.013256    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eed32ff863dd4870e3740d5321b1a4b3faa8299c8cdb089248659f8662552483"
Nov 01 12:39:34.013641 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.013184    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"
Nov 01 12:39:34.015040 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.015017    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"
Nov 01 12:39:34.016154 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.016117    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v" event={"ID":"437039d8-3fa5-4b26-b207-145ca139a447","Type":"ContainerDied","Data":"909ac2181e3b069044089e09ef81dcc03c6c2f81cc3cc4741086129134a96203"}
Nov 01 12:39:34.016242 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.016174    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="909ac2181e3b069044089e09ef81dcc03c6c2f81cc3cc4741086129134a96203"
Nov 01 12:39:34.185464 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.185436    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:34.324620 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.324532    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-openshift-ca\") pod \"179f214c-6ed2-40dd-bda9-51841f2125a9\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") "
Nov 01 12:39:34.324757 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.324630    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-xks-ca\") pod \"179f214c-6ed2-40dd-bda9-51841f2125a9\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") "
Nov 01 12:39:34.324807 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.324780    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179f214c-6ed2-40dd-bda9-51841f2125a9-kube-api-access-tcdm7\" (UniqueName: \"kubernetes.io/projected/179f214c-6ed2-40dd-bda9-51841f2125a9-kube-api-access-tcdm7\") pod \"179f214c-6ed2-40dd-bda9-51841f2125a9\" (UID: \"179f214c-6ed2-40dd-bda9-51841f2125a9\") "
Nov 01 12:39:34.324905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.324874    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-openshift-ca" pod "179f214c-6ed2-40dd-bda9-51841f2125a9" (UID: "179f214c-6ed2-40dd-bda9-51841f2125a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:34.325288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.325262    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:34.326742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.326714    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-xks-ca" pod "179f214c-6ed2-40dd-bda9-51841f2125a9" (UID: "179f214c-6ed2-40dd-bda9-51841f2125a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:34.326832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.326817    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179f214c-6ed2-40dd-bda9-51841f2125a9-kube-api-access-tcdm7" pod "179f214c-6ed2-40dd-bda9-51841f2125a9" (UID: "179f214c-6ed2-40dd-bda9-51841f2125a9"). InnerVolumeSpecName "kube-api-access-tcdm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:34.426155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.426119    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179f214c-6ed2-40dd-bda9-51841f2125a9-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:34.426155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:34.426148    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcdm7\" (UniqueName: \"kubernetes.io/projected/179f214c-6ed2-40dd-bda9-51841f2125a9-kube-api-access-tcdm7\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:35.023899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.023868    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"
Nov 01 12:39:35.024324 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.023909    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk" event={"ID":"179f214c-6ed2-40dd-bda9-51841f2125a9","Type":"ContainerDied","Data":"2c5a4c5a81ddbab3e1c1cbd636b3d6a2fbdb7c7c987e3f954fef875f00c1804a"}
Nov 01 12:39:35.024324 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.023967    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5a4c5a81ddbab3e1c1cbd636b3d6a2fbdb7c7c987e3f954fef875f00c1804a"
Nov 01 12:39:35.510300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.510263    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"]
Nov 01 12:39:35.511003 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.510984    5725 cpu_manager.go:414] "containerMap: removing container" podUID="437039d8-3fa5-4b26-b207-145ca139a447" containerName="revoke-keys"
Nov 01 12:39:35.511070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.511007    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437039d8-3fa5-4b26-b207-145ca139a447" containerName="revoke-keys"
Nov 01 12:39:35.511070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.511025    5725 cpu_manager.go:414] "containerMap: removing container" podUID="179f214c-6ed2-40dd-bda9-51841f2125a9" containerName="revoke-keys"
Nov 01 12:39:35.511070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.511030    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179f214c-6ed2-40dd-bda9-51841f2125a9" containerName="revoke-keys"
Nov 01 12:39:35.511070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.511067    5725 cpu_manager.go:414] "containerMap: removing container" podUID="7e9ec72c-dcb2-4e39-8d8e-014287745a36" containerName="revoke-keys"
Nov 01 12:39:35.511197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.511075    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9ec72c-dcb2-4e39-8d8e-014287745a36" containerName="revoke-keys"
Nov 01 12:39:35.511197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.511164    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="437039d8-3fa5-4b26-b207-145ca139a447" containerName="revoke-keys"
Nov 01 12:39:35.511197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.511174    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="179f214c-6ed2-40dd-bda9-51841f2125a9" containerName="revoke-keys"
Nov 01 12:39:35.511197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.511183    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9ec72c-dcb2-4e39-8d8e-014287745a36" containerName="revoke-keys"
Nov 01 12:39:35.514624 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.514604    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.564508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.564469    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"]
Nov 01 12:39:35.566585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.566561    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:39:35.566746 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.566612    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:39:35.639348 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.639315    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b133beba-ghrpk\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.639536 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.639387    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b133beba-ghrpk\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.639604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.639578    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5c6d\" (UniqueName: \"kubernetes.io/projected/4561a347-386c-43a4-8163-65e47df1fd34-kube-api-access-h5c6d\") pod \"maas-api-revoke-sub-b133beba-ghrpk\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.740581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.740538    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5c6d\" (UniqueName: \"kubernetes.io/projected/4561a347-386c-43a4-8163-65e47df1fd34-kube-api-access-h5c6d\") pod \"maas-api-revoke-sub-b133beba-ghrpk\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.740780 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.740606    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b133beba-ghrpk\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.740780 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.740662    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b133beba-ghrpk\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.741378 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.741344    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b133beba-ghrpk\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.743138 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.743112    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b133beba-ghrpk\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.794638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.794555    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5c6d\" (UniqueName: \"kubernetes.io/projected/4561a347-386c-43a4-8163-65e47df1fd34-kube-api-access-h5c6d\") pod \"maas-api-revoke-sub-b133beba-ghrpk\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:35.829426 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:35.829390    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:36.205874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:36.205787    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"]
Nov 01 12:39:36.212099 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:36.212067    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4561a347_386c_43a4_8163_65e47df1fd34.slice/crio-f475d7b4e408ac5bed5fcaa13fcdd0301d84d59e14984753b2f05d55de92cd35 WatchSource:0}: Error finding container f475d7b4e408ac5bed5fcaa13fcdd0301d84d59e14984753b2f05d55de92cd35: Status 404 returned error can't find the container with id f475d7b4e408ac5bed5fcaa13fcdd0301d84d59e14984753b2f05d55de92cd35
Nov 01 12:39:37.046548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:37.046495    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk" event={"ID":"4561a347-386c-43a4-8163-65e47df1fd34","Type":"ContainerStarted","Data":"97a5de8fa283c4c42414d7fcb39544d1cc84a74c71ded0b437f6331968616062"}
Nov 01 12:39:37.046766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:37.046575    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk" event={"ID":"4561a347-386c-43a4-8163-65e47df1fd34","Type":"ContainerStarted","Data":"f475d7b4e408ac5bed5fcaa13fcdd0301d84d59e14984753b2f05d55de92cd35"}
Nov 01 12:39:37.155450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:37.155385    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk" podStartSLOduration=2.155366798 podStartE2EDuration="2.155366798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:37.152364046 +0000 UTC m=+1018.659555261" watchObservedRunningTime="2026-11-01 12:39:37.155366798 +0000 UTC m=+1018.662558016"
Nov 01 12:39:38.056839 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:38.056803    5725 generic.go:356] "Generic (PLEG): container finished" podID="4561a347-386c-43a4-8163-65e47df1fd34" containerID="97a5de8fa283c4c42414d7fcb39544d1cc84a74c71ded0b437f6331968616062" exitCode=0
Nov 01 12:39:38.057377 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:38.056870    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk" event={"ID":"4561a347-386c-43a4-8163-65e47df1fd34","Type":"ContainerDied","Data":"97a5de8fa283c4c42414d7fcb39544d1cc84a74c71ded0b437f6331968616062"}
Nov 01 12:39:39.246395 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.246367    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:39.383621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.383530    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-openshift-ca\") pod \"4561a347-386c-43a4-8163-65e47df1fd34\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") "
Nov 01 12:39:39.383621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.383582    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4561a347-386c-43a4-8163-65e47df1fd34-kube-api-access-h5c6d\" (UniqueName: \"kubernetes.io/projected/4561a347-386c-43a4-8163-65e47df1fd34-kube-api-access-h5c6d\") pod \"4561a347-386c-43a4-8163-65e47df1fd34\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") "
Nov 01 12:39:39.383870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.383630    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-xks-ca\") pod \"4561a347-386c-43a4-8163-65e47df1fd34\" (UID: \"4561a347-386c-43a4-8163-65e47df1fd34\") "
Nov 01 12:39:39.383971 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.383943    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-openshift-ca" pod "4561a347-386c-43a4-8163-65e47df1fd34" (UID: "4561a347-386c-43a4-8163-65e47df1fd34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:39.384169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.384149    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:39.386669 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.386635    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4561a347-386c-43a4-8163-65e47df1fd34-kube-api-access-h5c6d" pod "4561a347-386c-43a4-8163-65e47df1fd34" (UID: "4561a347-386c-43a4-8163-65e47df1fd34"). InnerVolumeSpecName "kube-api-access-h5c6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:39.389166 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.389141    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-xks-ca" pod "4561a347-386c-43a4-8163-65e47df1fd34" (UID: "4561a347-386c-43a4-8163-65e47df1fd34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:39.485805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.485768    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5c6d\" (UniqueName: \"kubernetes.io/projected/4561a347-386c-43a4-8163-65e47df1fd34-kube-api-access-h5c6d\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:39.485805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:39.485799    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4561a347-386c-43a4-8163-65e47df1fd34-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:40.078874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:40.078831    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk" event={"ID":"4561a347-386c-43a4-8163-65e47df1fd34","Type":"ContainerDied","Data":"f475d7b4e408ac5bed5fcaa13fcdd0301d84d59e14984753b2f05d55de92cd35"}
Nov 01 12:39:40.078874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:40.078862    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"
Nov 01 12:39:40.079194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:40.078900    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f475d7b4e408ac5bed5fcaa13fcdd0301d84d59e14984753b2f05d55de92cd35"
Nov 01 12:39:41.129074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.125986    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"]
Nov 01 12:39:41.129074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.126816    5725 cpu_manager.go:414] "containerMap: removing container" podUID="4561a347-386c-43a4-8163-65e47df1fd34" containerName="revoke-keys"
Nov 01 12:39:41.129074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.126836    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4561a347-386c-43a4-8163-65e47df1fd34" containerName="revoke-keys"
Nov 01 12:39:41.129074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.127023    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="4561a347-386c-43a4-8163-65e47df1fd34" containerName="revoke-keys"
Nov 01 12:39:41.317453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.317416    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"]
Nov 01 12:39:41.317631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.317544    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.391030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.390932    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:39:41.392286 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.392265    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:39:41.456804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.456771    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8449121a-e767-4186-a689-0a89692a6d81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe26baf1-zz855\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.457022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.457009    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8449121a-e767-4186-a689-0a89692a6d81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe26baf1-zz855\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.457183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.457171    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblzq\" (UniqueName: \"kubernetes.io/projected/8449121a-e767-4186-a689-0a89692a6d81-kube-api-access-zblzq\") pod \"maas-api-revoke-sub-fe26baf1-zz855\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.557885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.557850    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8449121a-e767-4186-a689-0a89692a6d81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe26baf1-zz855\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.558091 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.557912    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblzq\" (UniqueName: \"kubernetes.io/projected/8449121a-e767-4186-a689-0a89692a6d81-kube-api-access-zblzq\") pod \"maas-api-revoke-sub-fe26baf1-zz855\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.558091 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.558013    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8449121a-e767-4186-a689-0a89692a6d81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe26baf1-zz855\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.558604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.558580    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8449121a-e767-4186-a689-0a89692a6d81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe26baf1-zz855\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.560417 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.560388    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8449121a-e767-4186-a689-0a89692a6d81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe26baf1-zz855\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.611499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.611464    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblzq\" (UniqueName: \"kubernetes.io/projected/8449121a-e767-4186-a689-0a89692a6d81-kube-api-access-zblzq\") pod \"maas-api-revoke-sub-fe26baf1-zz855\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:41.641534 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:41.641448    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:42.019898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:42.019854    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"]
Nov 01 12:39:42.026114 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:42.026078    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8449121a_e767_4186_a689_0a89692a6d81.slice/crio-812d7fee6bbb9221092a42b9f01feada0b359acd4eb48451469e38dcce709988 WatchSource:0}: Error finding container 812d7fee6bbb9221092a42b9f01feada0b359acd4eb48451469e38dcce709988: Status 404 returned error can't find the container with id 812d7fee6bbb9221092a42b9f01feada0b359acd4eb48451469e38dcce709988
Nov 01 12:39:42.117507 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:42.117472    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855" event={"ID":"8449121a-e767-4186-a689-0a89692a6d81","Type":"ContainerStarted","Data":"812d7fee6bbb9221092a42b9f01feada0b359acd4eb48451469e38dcce709988"}
Nov 01 12:39:43.125998 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:43.125957    5725 generic.go:356] "Generic (PLEG): container finished" podID="8449121a-e767-4186-a689-0a89692a6d81" containerID="48bd65435d9decf1d5365545a197467c5912903e10482b0e54c39d25e44c70d0" exitCode=0
Nov 01 12:39:43.126445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:43.126101    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855" event={"ID":"8449121a-e767-4186-a689-0a89692a6d81","Type":"ContainerDied","Data":"48bd65435d9decf1d5365545a197467c5912903e10482b0e54c39d25e44c70d0"}
Nov 01 12:39:44.405950 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.405876    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:44.503532 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.503499    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8449121a-e767-4186-a689-0a89692a6d81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8449121a-e767-4186-a689-0a89692a6d81-maas-api-openshift-ca\") pod \"8449121a-e767-4186-a689-0a89692a6d81\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") "
Nov 01 12:39:44.503722 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.503585    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8449121a-e767-4186-a689-0a89692a6d81-kube-api-access-zblzq\" (UniqueName: \"kubernetes.io/projected/8449121a-e767-4186-a689-0a89692a6d81-kube-api-access-zblzq\") pod \"8449121a-e767-4186-a689-0a89692a6d81\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") "
Nov 01 12:39:44.503811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.503789    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8449121a-e767-4186-a689-0a89692a6d81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8449121a-e767-4186-a689-0a89692a6d81-maas-api-xks-ca\") pod \"8449121a-e767-4186-a689-0a89692a6d81\" (UID: \"8449121a-e767-4186-a689-0a89692a6d81\") "
Nov 01 12:39:44.503906 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.503878    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8449121a-e767-4186-a689-0a89692a6d81-maas-api-openshift-ca" pod "8449121a-e767-4186-a689-0a89692a6d81" (UID: "8449121a-e767-4186-a689-0a89692a6d81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:44.504228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.504203    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8449121a-e767-4186-a689-0a89692a6d81-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:44.505772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.505748    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8449121a-e767-4186-a689-0a89692a6d81-kube-api-access-zblzq" pod "8449121a-e767-4186-a689-0a89692a6d81" (UID: "8449121a-e767-4186-a689-0a89692a6d81"). InnerVolumeSpecName "kube-api-access-zblzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:44.505872 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.505774    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8449121a-e767-4186-a689-0a89692a6d81-maas-api-xks-ca" pod "8449121a-e767-4186-a689-0a89692a6d81" (UID: "8449121a-e767-4186-a689-0a89692a6d81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:44.604953 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.604918    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8449121a-e767-4186-a689-0a89692a6d81-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:44.604953 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:44.604947    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblzq\" (UniqueName: \"kubernetes.io/projected/8449121a-e767-4186-a689-0a89692a6d81-kube-api-access-zblzq\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:45.147128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:45.147086    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855" event={"ID":"8449121a-e767-4186-a689-0a89692a6d81","Type":"ContainerDied","Data":"812d7fee6bbb9221092a42b9f01feada0b359acd4eb48451469e38dcce709988"}
Nov 01 12:39:45.147312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:45.147129    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"
Nov 01 12:39:45.147312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:45.147155    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="812d7fee6bbb9221092a42b9f01feada0b359acd4eb48451469e38dcce709988"
Nov 01 12:39:46.782156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.782111    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"]
Nov 01 12:39:46.782836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.782816    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8449121a-e767-4186-a689-0a89692a6d81" containerName="revoke-keys"
Nov 01 12:39:46.782882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.782838    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8449121a-e767-4186-a689-0a89692a6d81" containerName="revoke-keys"
Nov 01 12:39:46.782973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.782963    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8449121a-e767-4186-a689-0a89692a6d81" containerName="revoke-keys"
Nov 01 12:39:46.786593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.786573    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:46.832016 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.831975    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc67b\" (UniqueName: \"kubernetes.io/projected/d288684f-59f6-42ba-a058-0404dfba1d12-kube-api-access-lc67b\") pod \"maas-api-revoke-sub-159909d0-2ct68\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:46.832256 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.832162    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-159909d0-2ct68\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:46.832323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.832256    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-159909d0-2ct68\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:46.835914 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.835888    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"]
Nov 01 12:39:46.878624 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.878592    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:39:46.878794 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.878632    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:39:46.933923 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.933884    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc67b\" (UniqueName: \"kubernetes.io/projected/d288684f-59f6-42ba-a058-0404dfba1d12-kube-api-access-lc67b\") pod \"maas-api-revoke-sub-159909d0-2ct68\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:46.934169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.933999    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-159909d0-2ct68\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:46.934169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.934099    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-159909d0-2ct68\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:46.934994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.934963    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-159909d0-2ct68\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:46.939751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.937651    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-159909d0-2ct68\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:46.987357 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:46.987321    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc67b\" (UniqueName: \"kubernetes.io/projected/d288684f-59f6-42ba-a058-0404dfba1d12-kube-api-access-lc67b\") pod \"maas-api-revoke-sub-159909d0-2ct68\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:47.101885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:47.101798    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:47.501548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:47.501509    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"]
Nov 01 12:39:47.508208 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:47.508175    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd288684f_59f6_42ba_a058_0404dfba1d12.slice/crio-9443216cacbf7416afb261b615e08acd62b197025b01973f4ab7800cc9f0e5e0 WatchSource:0}: Error finding container 9443216cacbf7416afb261b615e08acd62b197025b01973f4ab7800cc9f0e5e0: Status 404 returned error can't find the container with id 9443216cacbf7416afb261b615e08acd62b197025b01973f4ab7800cc9f0e5e0
Nov 01 12:39:48.172767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:48.172681    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68" event={"ID":"d288684f-59f6-42ba-a058-0404dfba1d12","Type":"ContainerStarted","Data":"db51a2e2756847f6dee0e44fc488982f432b90685042e13a5e9bac1b30bf3092"}
Nov 01 12:39:48.172767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:48.172744    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68" event={"ID":"d288684f-59f6-42ba-a058-0404dfba1d12","Type":"ContainerStarted","Data":"9443216cacbf7416afb261b615e08acd62b197025b01973f4ab7800cc9f0e5e0"}
Nov 01 12:39:48.282308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:48.282255    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68" podStartSLOduration=2.2822410619999998 podStartE2EDuration="2.282241062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:48.27889359 +0000 UTC m=+1029.786084833" watchObservedRunningTime="2026-11-01 12:39:48.282241062 +0000 UTC m=+1029.789432320"
Nov 01 12:39:49.181460 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:49.181426    5725 generic.go:356] "Generic (PLEG): container finished" podID="d288684f-59f6-42ba-a058-0404dfba1d12" containerID="db51a2e2756847f6dee0e44fc488982f432b90685042e13a5e9bac1b30bf3092" exitCode=0
Nov 01 12:39:49.181872 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:49.181509    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68" event={"ID":"d288684f-59f6-42ba-a058-0404dfba1d12","Type":"ContainerDied","Data":"db51a2e2756847f6dee0e44fc488982f432b90685042e13a5e9bac1b30bf3092"}
Nov 01 12:39:50.369851 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.369816    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:50.487470 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.487423    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d288684f-59f6-42ba-a058-0404dfba1d12-kube-api-access-lc67b\" (UniqueName: \"kubernetes.io/projected/d288684f-59f6-42ba-a058-0404dfba1d12-kube-api-access-lc67b\") pod \"d288684f-59f6-42ba-a058-0404dfba1d12\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") "
Nov 01 12:39:50.487659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.487585    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-xks-ca\") pod \"d288684f-59f6-42ba-a058-0404dfba1d12\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") "
Nov 01 12:39:50.487734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.487677    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-openshift-ca\") pod \"d288684f-59f6-42ba-a058-0404dfba1d12\" (UID: \"d288684f-59f6-42ba-a058-0404dfba1d12\") "
Nov 01 12:39:50.488242 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.488211    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-openshift-ca" pod "d288684f-59f6-42ba-a058-0404dfba1d12" (UID: "d288684f-59f6-42ba-a058-0404dfba1d12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:50.490189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.490160    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d288684f-59f6-42ba-a058-0404dfba1d12-kube-api-access-lc67b" pod "d288684f-59f6-42ba-a058-0404dfba1d12" (UID: "d288684f-59f6-42ba-a058-0404dfba1d12"). InnerVolumeSpecName "kube-api-access-lc67b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:50.490335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.490166    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-xks-ca" pod "d288684f-59f6-42ba-a058-0404dfba1d12" (UID: "d288684f-59f6-42ba-a058-0404dfba1d12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:50.589732 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.589698    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc67b\" (UniqueName: \"kubernetes.io/projected/d288684f-59f6-42ba-a058-0404dfba1d12-kube-api-access-lc67b\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:50.589732 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.589731    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:50.589934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:50.589743    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d288684f-59f6-42ba-a058-0404dfba1d12-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:51.199509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:51.199478    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"
Nov 01 12:39:51.199688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:51.199521    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68" event={"ID":"d288684f-59f6-42ba-a058-0404dfba1d12","Type":"ContainerDied","Data":"9443216cacbf7416afb261b615e08acd62b197025b01973f4ab7800cc9f0e5e0"}
Nov 01 12:39:51.199688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:51.199574    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9443216cacbf7416afb261b615e08acd62b197025b01973f4ab7800cc9f0e5e0"
Nov 01 12:39:55.128919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.128885    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"]
Nov 01 12:39:55.129425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.129410    5725 cpu_manager.go:414] "containerMap: removing container" podUID="d288684f-59f6-42ba-a058-0404dfba1d12" containerName="revoke-keys"
Nov 01 12:39:55.129488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.129426    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d288684f-59f6-42ba-a058-0404dfba1d12" containerName="revoke-keys"
Nov 01 12:39:55.129553 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.129543    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="d288684f-59f6-42ba-a058-0404dfba1d12" containerName="revoke-keys"
Nov 01 12:39:55.133196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.133176    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.184156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.184121    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"]
Nov 01 12:39:55.208934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.208901    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:39:55.208934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.208906    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:39:55.243149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.243119    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6ef091-x284p\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.243409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.243270    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6ef091-x284p\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.243409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.243371    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwt4t\" (UniqueName: \"kubernetes.io/projected/9b902290-a726-4b0d-9993-b2c2b015dd57-kube-api-access-bwt4t\") pod \"maas-api-revoke-sub-4c6ef091-x284p\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.344447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.344404    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwt4t\" (UniqueName: \"kubernetes.io/projected/9b902290-a726-4b0d-9993-b2c2b015dd57-kube-api-access-bwt4t\") pod \"maas-api-revoke-sub-4c6ef091-x284p\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.344604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.344540    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6ef091-x284p\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.344677 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.344654    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6ef091-x284p\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.345492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.345461    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c6ef091-x284p\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.347288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.347260    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c6ef091-x284p\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.398608 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.398535    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwt4t\" (UniqueName: \"kubernetes.io/projected/9b902290-a726-4b0d-9993-b2c2b015dd57-kube-api-access-bwt4t\") pod \"maas-api-revoke-sub-4c6ef091-x284p\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.449409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.449371    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:55.830942 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:55.830908    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"]
Nov 01 12:39:55.836549 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:55.836522    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b902290_a726_4b0d_9993_b2c2b015dd57.slice/crio-63a75717d717570fff485f6901ae1c0858b1bcf3ffecb69f72fc0d742987602a WatchSource:0}: Error finding container 63a75717d717570fff485f6901ae1c0858b1bcf3ffecb69f72fc0d742987602a: Status 404 returned error can't find the container with id 63a75717d717570fff485f6901ae1c0858b1bcf3ffecb69f72fc0d742987602a
Nov 01 12:39:56.252553 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.252520    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p" event={"ID":"9b902290-a726-4b0d-9993-b2c2b015dd57","Type":"ContainerStarted","Data":"6c5263653d082673a48184b596fa528db0d1ef817c27758a69b9094d4814eb50"}
Nov 01 12:39:56.252997 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.252579    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p" event={"ID":"9b902290-a726-4b0d-9993-b2c2b015dd57","Type":"ContainerStarted","Data":"63a75717d717570fff485f6901ae1c0858b1bcf3ffecb69f72fc0d742987602a"}
Nov 01 12:39:56.281065 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.281020    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"]
Nov 01 12:39:56.285224 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.285200    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.338451 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.338414    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"]
Nov 01 12:39:56.356142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.356099    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9zjf\" (UniqueName: \"kubernetes.io/projected/f41e6112-9475-4ee9-9d6b-8437cebddc96-kube-api-access-r9zjf\") pod \"maas-api-revoke-sub-c0dadb92-knkgt\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.356323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.356183    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0dadb92-knkgt\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.356323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.356232    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0dadb92-knkgt\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.384304 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.384252    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p" podStartSLOduration=1.384238681 podStartE2EDuration="1.384238681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:56.380624429 +0000 UTC m=+1037.887815645" watchObservedRunningTime="2026-11-01 12:39:56.384238681 +0000 UTC m=+1037.891429899"
Nov 01 12:39:56.458019 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.457918    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zjf\" (UniqueName: \"kubernetes.io/projected/f41e6112-9475-4ee9-9d6b-8437cebddc96-kube-api-access-r9zjf\") pod \"maas-api-revoke-sub-c0dadb92-knkgt\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.458019 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.457972    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0dadb92-knkgt\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.458263 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.458026    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0dadb92-knkgt\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.458868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.458842    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0dadb92-knkgt\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.460650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.460626    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0dadb92-knkgt\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.511872 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.511842    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9zjf\" (UniqueName: \"kubernetes.io/projected/f41e6112-9475-4ee9-9d6b-8437cebddc96-kube-api-access-r9zjf\") pod \"maas-api-revoke-sub-c0dadb92-knkgt\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.599221 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.599189    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:56.983243 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:56.983211    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"]
Nov 01 12:39:56.986785 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:39:56.986756    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41e6112_9475_4ee9_9d6b_8437cebddc96.slice/crio-af05f2164fc371e59d05048fed3c1598ad626a30ee694db0a84fb0a9e0041edc WatchSource:0}: Error finding container af05f2164fc371e59d05048fed3c1598ad626a30ee694db0a84fb0a9e0041edc: Status 404 returned error can't find the container with id af05f2164fc371e59d05048fed3c1598ad626a30ee694db0a84fb0a9e0041edc
Nov 01 12:39:57.262976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:57.262941    5725 generic.go:356] "Generic (PLEG): container finished" podID="9b902290-a726-4b0d-9993-b2c2b015dd57" containerID="6c5263653d082673a48184b596fa528db0d1ef817c27758a69b9094d4814eb50" exitCode=0
Nov 01 12:39:57.263420 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:57.263033    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p" event={"ID":"9b902290-a726-4b0d-9993-b2c2b015dd57","Type":"ContainerDied","Data":"6c5263653d082673a48184b596fa528db0d1ef817c27758a69b9094d4814eb50"}
Nov 01 12:39:57.264902 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:57.264875    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt" event={"ID":"f41e6112-9475-4ee9-9d6b-8437cebddc96","Type":"ContainerStarted","Data":"a3d5a8d06152da071b4a41b4b4056d4b58889b4bd2ff7ca7a0ee7e2609980cf8"}
Nov 01 12:39:57.265044 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:57.264920    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt" event={"ID":"f41e6112-9475-4ee9-9d6b-8437cebddc96","Type":"ContainerStarted","Data":"af05f2164fc371e59d05048fed3c1598ad626a30ee694db0a84fb0a9e0041edc"}
Nov 01 12:39:57.479078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:57.479009    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt" podStartSLOduration=1.478994165 podStartE2EDuration="1.478994165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:57.47886202 +0000 UTC m=+1038.986053219" watchObservedRunningTime="2026-11-01 12:39:57.478994165 +0000 UTC m=+1038.986185382"
Nov 01 12:39:58.275674 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.275636    5725 generic.go:356] "Generic (PLEG): container finished" podID="f41e6112-9475-4ee9-9d6b-8437cebddc96" containerID="a3d5a8d06152da071b4a41b4b4056d4b58889b4bd2ff7ca7a0ee7e2609980cf8" exitCode=0
Nov 01 12:39:58.276104 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.275715    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt" event={"ID":"f41e6112-9475-4ee9-9d6b-8437cebddc96","Type":"ContainerDied","Data":"a3d5a8d06152da071b4a41b4b4056d4b58889b4bd2ff7ca7a0ee7e2609980cf8"}
Nov 01 12:39:58.439835 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.439805    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:58.583011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.582923    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b902290-a726-4b0d-9993-b2c2b015dd57-kube-api-access-bwt4t\" (UniqueName: \"kubernetes.io/projected/9b902290-a726-4b0d-9993-b2c2b015dd57-kube-api-access-bwt4t\") pod \"9b902290-a726-4b0d-9993-b2c2b015dd57\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") "
Nov 01 12:39:58.583011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.582993    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-xks-ca\") pod \"9b902290-a726-4b0d-9993-b2c2b015dd57\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") "
Nov 01 12:39:58.583206 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.583158    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-openshift-ca\") pod \"9b902290-a726-4b0d-9993-b2c2b015dd57\" (UID: \"9b902290-a726-4b0d-9993-b2c2b015dd57\") "
Nov 01 12:39:58.583779 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.583737    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-openshift-ca" pod "9b902290-a726-4b0d-9993-b2c2b015dd57" (UID: "9b902290-a726-4b0d-9993-b2c2b015dd57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:58.585257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.585235    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-xks-ca" pod "9b902290-a726-4b0d-9993-b2c2b015dd57" (UID: "9b902290-a726-4b0d-9993-b2c2b015dd57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:58.585341 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.585235    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b902290-a726-4b0d-9993-b2c2b015dd57-kube-api-access-bwt4t" pod "9b902290-a726-4b0d-9993-b2c2b015dd57" (UID: "9b902290-a726-4b0d-9993-b2c2b015dd57"). InnerVolumeSpecName "kube-api-access-bwt4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:58.684781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.684741    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwt4t\" (UniqueName: \"kubernetes.io/projected/9b902290-a726-4b0d-9993-b2c2b015dd57-kube-api-access-bwt4t\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:58.684781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.684775    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:58.684781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:58.684789    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b902290-a726-4b0d-9993-b2c2b015dd57-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:59.284335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.284301    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"
Nov 01 12:39:59.284777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.284334    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p" event={"ID":"9b902290-a726-4b0d-9993-b2c2b015dd57","Type":"ContainerDied","Data":"63a75717d717570fff485f6901ae1c0858b1bcf3ffecb69f72fc0d742987602a"}
Nov 01 12:39:59.284777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.284421    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63a75717d717570fff485f6901ae1c0858b1bcf3ffecb69f72fc0d742987602a"
Nov 01 12:39:59.441040 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.441013    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:39:59.531482 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.531446    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"]
Nov 01 12:39:59.532018 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.531994    5725 cpu_manager.go:414] "containerMap: removing container" podUID="9b902290-a726-4b0d-9993-b2c2b015dd57" containerName="revoke-keys"
Nov 01 12:39:59.532018 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.532013    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b902290-a726-4b0d-9993-b2c2b015dd57" containerName="revoke-keys"
Nov 01 12:39:59.532248 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.532036    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f41e6112-9475-4ee9-9d6b-8437cebddc96" containerName="revoke-keys"
Nov 01 12:39:59.532248 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.532043    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41e6112-9475-4ee9-9d6b-8437cebddc96" containerName="revoke-keys"
Nov 01 12:39:59.532248 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.532192    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41e6112-9475-4ee9-9d6b-8437cebddc96" containerName="revoke-keys"
Nov 01 12:39:59.532248 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.532209    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b902290-a726-4b0d-9993-b2c2b015dd57" containerName="revoke-keys"
Nov 01 12:39:59.536156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.536093    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.594735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.594696    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-xks-ca\") pod \"f41e6112-9475-4ee9-9d6b-8437cebddc96\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") "
Nov 01 12:39:59.594961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.594777    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41e6112-9475-4ee9-9d6b-8437cebddc96-kube-api-access-r9zjf\" (UniqueName: \"kubernetes.io/projected/f41e6112-9475-4ee9-9d6b-8437cebddc96-kube-api-access-r9zjf\") pod \"f41e6112-9475-4ee9-9d6b-8437cebddc96\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") "
Nov 01 12:39:59.594961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.594948    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-openshift-ca\") pod \"f41e6112-9475-4ee9-9d6b-8437cebddc96\" (UID: \"f41e6112-9475-4ee9-9d6b-8437cebddc96\") "
Nov 01 12:39:59.595387 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.595359    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-openshift-ca" pod "f41e6112-9475-4ee9-9d6b-8437cebddc96" (UID: "f41e6112-9475-4ee9-9d6b-8437cebddc96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:39:59.601651 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.598535    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41e6112-9475-4ee9-9d6b-8437cebddc96-kube-api-access-r9zjf" pod "f41e6112-9475-4ee9-9d6b-8437cebddc96" (UID: "f41e6112-9475-4ee9-9d6b-8437cebddc96"). InnerVolumeSpecName "kube-api-access-r9zjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:39:59.601651 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.601114    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-xks-ca" pod "f41e6112-9475-4ee9-9d6b-8437cebddc96" (UID: "f41e6112-9475-4ee9-9d6b-8437cebddc96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:39:59.660074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.659415    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"]
Nov 01 12:39:59.696810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.696764    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.696981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.696873    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.696981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.696932    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.696981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.696951    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.697116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.696983    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b06a90-c616-4913-9a07-52eec95b9826-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.697162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.697107    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.697200 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.697172    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b06a90-c616-4913-9a07-52eec95b9826-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.697243 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.697233    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zld\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-kube-api-access-d9zld\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.697348 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.697331    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.697462 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.697442    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:59.697513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.697470    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f41e6112-9475-4ee9-9d6b-8437cebddc96-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:59.697513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.697488    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9zjf\" (UniqueName: \"kubernetes.io/projected/f41e6112-9475-4ee9-9d6b-8437cebddc96-kube-api-access-r9zjf\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:39:59.798631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.798548    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zld\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-kube-api-access-d9zld\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.798631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.798594    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.798880 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.798636    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.798880 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.798664    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.798880 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.798705    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.798880 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.798745    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.798880 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.798795    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b06a90-c616-4913-9a07-52eec95b9826-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.798880 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.798823    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.798880 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.798868    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b06a90-c616-4913-9a07-52eec95b9826-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.799259 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.799140    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.799259 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.799204    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.799259 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.799233    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.799403 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.799282    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.799596 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.799577    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b06a90-c616-4913-9a07-52eec95b9826-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.801167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.801135    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.801379 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.801363    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b06a90-c616-4913-9a07-52eec95b9826-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.851964 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.851931    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:39:59.852137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:39:59.852098    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zld\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-kube-api-access-d9zld\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:40:00.152167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:00.152069    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:40:00.295651 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:00.295614    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"
Nov 01 12:40:00.296149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:00.295645    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt" event={"ID":"f41e6112-9475-4ee9-9d6b-8437cebddc96","Type":"ContainerDied","Data":"af05f2164fc371e59d05048fed3c1598ad626a30ee694db0a84fb0a9e0041edc"}
Nov 01 12:40:00.296149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:00.295894    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af05f2164fc371e59d05048fed3c1598ad626a30ee694db0a84fb0a9e0041edc"
Nov 01 12:40:00.685711 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:00.685671    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"]
Nov 01 12:40:00.726919 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:40:00.726708    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b06a90_c616_4913_9a07_52eec95b9826.slice/crio-012f64f62af08006ccfed7b7724666a6e5541c0d0975c714751d71d3d4326dd5 WatchSource:0}: Error finding container 012f64f62af08006ccfed7b7724666a6e5541c0d0975c714751d71d3d4326dd5: Status 404 returned error can't find the container with id 012f64f62af08006ccfed7b7724666a6e5541c0d0975c714751d71d3d4326dd5
Nov 01 12:40:00.745426 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:00.735736    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:40:00.745426 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:00.735843    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:40:00.745426 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:00.735916    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:40:01.049292 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:40:01.049219    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:40:01.355182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:01.354490    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" event={"ID":"22b06a90-c616-4913-9a07-52eec95b9826","Type":"ContainerStarted","Data":"3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39"}
Nov 01 12:40:01.355182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:01.354566    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" event={"ID":"22b06a90-c616-4913-9a07-52eec95b9826","Type":"ContainerStarted","Data":"012f64f62af08006ccfed7b7724666a6e5541c0d0975c714751d71d3d4326dd5"}
Nov 01 12:40:01.481931 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:01.481860    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podStartSLOduration=2.481842142 podStartE2EDuration="2.481842142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:01.478782191 +0000 UTC m=+1042.985973435" watchObservedRunningTime="2026-11-01 12:40:01.481842142 +0000 UTC m=+1042.989033359"
Nov 01 12:40:02.154911 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:02.154866    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:40:02.155122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:02.155009    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 12:40:02.155122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:02.155092    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 12:40:03.067072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:03.062689    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:40:03.154445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:03.154406    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 12:40:03.154754 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:03.154715    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 12:40:04.153074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:04.152514    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 12:40:04.153074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:04.152581    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 12:40:05.153077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:05.152460    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 12:40:05.153077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:05.152524    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 12:40:06.152974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:06.152502    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 12:40:06.152974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:06.152579    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 12:40:07.152512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:07.152474    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 12:40:07.152894 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:07.152865    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 12:40:08.153435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:08.153399    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 12:40:08.153854 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:08.153454    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 12:40:09.368752 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:09.368718    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:40:09.369328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:09.369262    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:40:09.370335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:09.370314    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:40:11.998591 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:11.998552    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"]
Nov 01 12:40:12.003896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.003873    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.054468 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.054432    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"]
Nov 01 12:40:12.068880 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.068847    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twqtz\" (UniqueName: \"kubernetes.io/projected/768d1620-f31c-4161-bb42-c8f3d27c24a6-kube-api-access-twqtz\") pod \"maas-api-revoke-sub-ee81217e-ljnsg\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.069076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.068918    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee81217e-ljnsg\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.069147 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.069090    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee81217e-ljnsg\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.096605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.096574    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:40:12.096757 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.096609    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:40:12.170634 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.170600    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee81217e-ljnsg\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.170812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.170656    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twqtz\" (UniqueName: \"kubernetes.io/projected/768d1620-f31c-4161-bb42-c8f3d27c24a6-kube-api-access-twqtz\") pod \"maas-api-revoke-sub-ee81217e-ljnsg\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.170812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.170696    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee81217e-ljnsg\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.171444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.171419    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee81217e-ljnsg\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.173180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.173163    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee81217e-ljnsg\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.224269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.224237    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twqtz\" (UniqueName: \"kubernetes.io/projected/768d1620-f31c-4161-bb42-c8f3d27c24a6-kube-api-access-twqtz\") pod \"maas-api-revoke-sub-ee81217e-ljnsg\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.318224 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.318133    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:12.693099 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:12.693007    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"]
Nov 01 12:40:12.698549 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:40:12.698520    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768d1620_f31c_4161_bb42_c8f3d27c24a6.slice/crio-d449eb37d8968de8884fa4562fe99db78f235e0ecbf85c11d1cf9a501d2606e7 WatchSource:0}: Error finding container d449eb37d8968de8884fa4562fe99db78f235e0ecbf85c11d1cf9a501d2606e7: Status 404 returned error can't find the container with id d449eb37d8968de8884fa4562fe99db78f235e0ecbf85c11d1cf9a501d2606e7
Nov 01 12:40:13.548363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:13.548316    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg" event={"ID":"768d1620-f31c-4161-bb42-c8f3d27c24a6","Type":"ContainerStarted","Data":"aed35533b27a29ef730cfc12e3995fad8e88b82706d4802924fa167a16171a3e"}
Nov 01 12:40:13.548758 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:13.548390    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg" event={"ID":"768d1620-f31c-4161-bb42-c8f3d27c24a6","Type":"ContainerStarted","Data":"d449eb37d8968de8884fa4562fe99db78f235e0ecbf85c11d1cf9a501d2606e7"}
Nov 01 12:40:13.657250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:13.657196    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg" podStartSLOduration=2.657181617 podStartE2EDuration="2.657181617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:13.655597628 +0000 UTC m=+1055.162793088" watchObservedRunningTime="2026-11-01 12:40:13.657181617 +0000 UTC m=+1055.164372836"
Nov 01 12:40:14.557082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.557034    5725 generic.go:356] "Generic (PLEG): container finished" podID="768d1620-f31c-4161-bb42-c8f3d27c24a6" containerID="aed35533b27a29ef730cfc12e3995fad8e88b82706d4802924fa167a16171a3e" exitCode=0
Nov 01 12:40:14.557524 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.557119    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg" event={"ID":"768d1620-f31c-4161-bb42-c8f3d27c24a6","Type":"ContainerDied","Data":"aed35533b27a29ef730cfc12e3995fad8e88b82706d4802924fa167a16171a3e"}
Nov 01 12:40:14.660992 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.660941    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"]
Nov 01 12:40:14.665311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.665292    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.714156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.714122    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"]
Nov 01 12:40:14.777439 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.777410    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9514b0-openshift-default-dockercfg-4cdz5\""
Nov 01 12:40:14.802300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.802262    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.802477 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.802415    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.802477 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.802462    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwnpb\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-kube-api-access-lwnpb\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.802605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.802484    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.802605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.802505    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.802605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.802545    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.802733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.802637    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbce536-40ca-49ee-b60d-0c2181800e4c-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.802733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.802695    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.802733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.802720    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.904441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904357    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.904441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904395    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwnpb\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-kube-api-access-lwnpb\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.904441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904412    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.904683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904533    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.904683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904570    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.904683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904595    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbce536-40ca-49ee-b60d-0c2181800e4c-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.904683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904635    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.904683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904664    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.904943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904784    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.905004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904938    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.905004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.904977    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.905179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.905157    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.905239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.905199    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.905444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.905422    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbce536-40ca-49ee-b60d-0c2181800e4c-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.906872 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.906842    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.907122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.907104    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.957955 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.957926    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.958125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.958008    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwnpb\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-kube-api-access-lwnpb\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:14.979944 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:14.979915    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:15.410702 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.410665    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"]
Nov 01 12:40:15.416248 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:40:15.416213    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbce536_40ca_49ee_b60d_0c2181800e4c.slice/crio-83d20241b1e0f3218eac4972f0a4e3c993ffcb6b1dda7ec70ce9e2762bc74b2f WatchSource:0}: Error finding container 83d20241b1e0f3218eac4972f0a4e3c993ffcb6b1dda7ec70ce9e2762bc74b2f: Status 404 returned error can't find the container with id 83d20241b1e0f3218eac4972f0a4e3c993ffcb6b1dda7ec70ce9e2762bc74b2f
Nov 01 12:40:15.418678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.418642    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:40:15.418776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.418710    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:40:15.418776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.418740    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:40:15.569276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.569208    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" event={"ID":"bbbce536-40ca-49ee-b60d-0c2181800e4c","Type":"ContainerStarted","Data":"a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a"}
Nov 01 12:40:15.569276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.569268    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" event={"ID":"bbbce536-40ca-49ee-b60d-0c2181800e4c","Type":"ContainerStarted","Data":"83d20241b1e0f3218eac4972f0a4e3c993ffcb6b1dda7ec70ce9e2762bc74b2f"}
Nov 01 12:40:15.682915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.682822    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" podStartSLOduration=1.682806513 podStartE2EDuration="1.682806513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:15.677937219 +0000 UTC m=+1057.185128438" watchObservedRunningTime="2026-11-01 12:40:15.682806513 +0000 UTC m=+1057.189997746"
Nov 01 12:40:15.709949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.709920    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:15.815083 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.815038    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768d1620-f31c-4161-bb42-c8f3d27c24a6-kube-api-access-twqtz\" (UniqueName: \"kubernetes.io/projected/768d1620-f31c-4161-bb42-c8f3d27c24a6-kube-api-access-twqtz\") pod \"768d1620-f31c-4161-bb42-c8f3d27c24a6\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") "
Nov 01 12:40:15.815274 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.815158    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-xks-ca\") pod \"768d1620-f31c-4161-bb42-c8f3d27c24a6\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") "
Nov 01 12:40:15.815347 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.815308    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-openshift-ca\") pod \"768d1620-f31c-4161-bb42-c8f3d27c24a6\" (UID: \"768d1620-f31c-4161-bb42-c8f3d27c24a6\") "
Nov 01 12:40:15.816037 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.816009    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-openshift-ca" pod "768d1620-f31c-4161-bb42-c8f3d27c24a6" (UID: "768d1620-f31c-4161-bb42-c8f3d27c24a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:40:15.819407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.819341    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768d1620-f31c-4161-bb42-c8f3d27c24a6-kube-api-access-twqtz" pod "768d1620-f31c-4161-bb42-c8f3d27c24a6" (UID: "768d1620-f31c-4161-bb42-c8f3d27c24a6"). InnerVolumeSpecName "kube-api-access-twqtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:40:15.820150 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.820122    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-xks-ca" pod "768d1620-f31c-4161-bb42-c8f3d27c24a6" (UID: "768d1620-f31c-4161-bb42-c8f3d27c24a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:40:15.917169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.917135    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twqtz\" (UniqueName: \"kubernetes.io/projected/768d1620-f31c-4161-bb42-c8f3d27c24a6-kube-api-access-twqtz\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:40:15.917169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.917163    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:40:15.917272 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.917175    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/768d1620-f31c-4161-bb42-c8f3d27c24a6-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:40:15.980981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:15.980924    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:16.578409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:16.578373    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"
Nov 01 12:40:16.578409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:16.578386    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg" event={"ID":"768d1620-f31c-4161-bb42-c8f3d27c24a6","Type":"ContainerDied","Data":"d449eb37d8968de8884fa4562fe99db78f235e0ecbf85c11d1cf9a501d2606e7"}
Nov 01 12:40:16.578848 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:16.578440    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d449eb37d8968de8884fa4562fe99db78f235e0ecbf85c11d1cf9a501d2606e7"
Nov 01 12:40:16.982405 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:16.982365    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 12:40:16.982581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:16.982426    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 12:40:17.980940 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:17.980895    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 12:40:17.981364 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:17.980960    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 12:40:18.980550 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:18.980512    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 12:40:18.980723 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:18.980588    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 12:40:19.980612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:19.980572    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 12:40:19.981035 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:19.980644    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 12:40:20.980293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:20.980257    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 01 12:40:20.980468 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:20.980315    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 01 12:40:22.892853 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:22.892563    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:22.894272 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:22.893469    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:22.894919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:22.894893    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 12:40:29.221685 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.221650    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rnj8v-hj8cd"]
Nov 01 12:40:29.222228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.222213    5725 cpu_manager.go:414] "containerMap: removing container" podUID="768d1620-f31c-4161-bb42-c8f3d27c24a6" containerName="revoke-keys"
Nov 01 12:40:29.222279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.222229    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768d1620-f31c-4161-bb42-c8f3d27c24a6" containerName="revoke-keys"
Nov 01 12:40:29.222350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.222333    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="768d1620-f31c-4161-bb42-c8f3d27c24a6" containerName="revoke-keys"
Nov 01 12:40:29.226197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.226176    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.279218 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.279186    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rnj8v-hj8cd"]
Nov 01 12:40:29.303994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.303958    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-service-ca-bundle\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.304243 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.304202    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pk2r\" (UniqueName: \"kubernetes.io/projected/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-kube-api-access-6pk2r\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.304320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.304252    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-trusted-ca-bundle\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.304400 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.304384    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-archives-path\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.406073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.406011    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pk2r\" (UniqueName: \"kubernetes.io/projected/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-kube-api-access-6pk2r\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.406241 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.406090    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-trusted-ca-bundle\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.406241 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.406163    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-archives-path\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.406241 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.406207    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-service-ca-bundle\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.406498 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.406479    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-archives-path\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.406976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.406952    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-service-ca-bundle\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.407012 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.406994    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-trusted-ca-bundle\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.460097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.460031    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pk2r\" (UniqueName: \"kubernetes.io/projected/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-kube-api-access-6pk2r\") pod \"periodic-gathering-rnj8v-hj8cd\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") " pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.542014 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.541914    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:40:29.626032 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.626001    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"]
Nov 01 12:40:29.631853 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.631828    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.684883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.684855    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"]
Nov 01 12:40:29.710718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.709945    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.710718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.709994    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.710718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.710014    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.710718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.710034    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.710718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.710109    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.710718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.710201    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.710718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.710220    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.710718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.710274    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61f02b5-3268-4b1a-b946-3c112ee12dec-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.710718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.710448    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdcrn\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-kube-api-access-mdcrn\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.812100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812005    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61f02b5-3268-4b1a-b946-3c112ee12dec-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.812100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812065    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcrn\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-kube-api-access-mdcrn\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.812638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812153    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.812638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812183    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.812638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812230    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.812638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812258    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.812638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812320    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.812638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812419    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.812638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812445    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.813004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812702    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.813004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812719    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.813004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812723    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61f02b5-3268-4b1a-b946-3c112ee12dec-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.813004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.812991    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.813235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.813088    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.814772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.814750    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.815023 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.815000    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.866300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.866264    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcrn\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-kube-api-access-mdcrn\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.866507 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.866484    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:29.925837 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.925797    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rnj8v-hj8cd"]
Nov 01 12:40:29.931268 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:40:29.931240    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c751f45_9cb5_4c77_b4c9_1c30ea327ff8.slice/crio-da0e221f666a47a91f0a708a5d2f26566a933f72cede0f214f7c37a135ed8da8 WatchSource:0}: Error finding container da0e221f666a47a91f0a708a5d2f26566a933f72cede0f214f7c37a135ed8da8: Status 404 returned error can't find the container with id da0e221f666a47a91f0a708a5d2f26566a933f72cede0f214f7c37a135ed8da8
Nov 01 12:40:29.950744 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:29.950719    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:30.386955 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.386920    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"]
Nov 01 12:40:30.391945 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:40:30.391919    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61f02b5_3268_4b1a_b946_3c112ee12dec.slice/crio-fdc49fcd872a6da0ae604dc62b0679a07027a3e8a476509b45e603abe7f48d65 WatchSource:0}: Error finding container fdc49fcd872a6da0ae604dc62b0679a07027a3e8a476509b45e603abe7f48d65: Status 404 returned error can't find the container with id fdc49fcd872a6da0ae604dc62b0679a07027a3e8a476509b45e603abe7f48d65
Nov 01 12:40:30.394365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.394332    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:40:30.394445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.394396    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:40:30.394445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.394425    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:40:30.696537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.696437    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" event={"ID":"c61f02b5-3268-4b1a-b946-3c112ee12dec","Type":"ContainerStarted","Data":"3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff"}
Nov 01 12:40:30.696698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.696554    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" event={"ID":"c61f02b5-3268-4b1a-b946-3c112ee12dec","Type":"ContainerStarted","Data":"fdc49fcd872a6da0ae604dc62b0679a07027a3e8a476509b45e603abe7f48d65"}
Nov 01 12:40:30.699465 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.699429    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rnj8v-hj8cd" event={"ID":"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8","Type":"ContainerStarted","Data":"527efbd7c1bcb39f00b1fa9a30c536cbe2c5a4cbfba6e932ea3b813776017d03"}
Nov 01 12:40:30.699610 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.699483    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rnj8v-hj8cd" event={"ID":"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8","Type":"ContainerStarted","Data":"da0e221f666a47a91f0a708a5d2f26566a933f72cede0f214f7c37a135ed8da8"}
Nov 01 12:40:30.809777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.809725    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" podStartSLOduration=1.809711342 podStartE2EDuration="1.809711342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:30.808327195 +0000 UTC m=+1072.315518444" watchObservedRunningTime="2026-11-01 12:40:30.809711342 +0000 UTC m=+1072.316902558"
Nov 01 12:40:30.919513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.919459    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rnj8v-hj8cd" podStartSLOduration=1.919444251 podStartE2EDuration="1.919444251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:30.914794072 +0000 UTC m=+1072.421985289" watchObservedRunningTime="2026-11-01 12:40:30.919444251 +0000 UTC m=+1072.426635467"
Nov 01 12:40:30.951383 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:30.951303    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:31.038989 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:40:31.038945    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:40:31.953075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:31.952793    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 12:40:31.953075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:31.952872    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 12:40:32.955073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:32.954425    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 12:40:32.955073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:32.954498    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 12:40:33.951948 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:33.951911    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 12:40:33.952273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:33.952245    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 12:40:34.952559 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:34.952519    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 12:40:34.953092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:34.952587    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 12:40:35.954185 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:35.954146    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 12:40:35.954768 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:35.954736    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 12:40:36.951420 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:36.951383    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 12:40:36.951604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:36.951456    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 12:40:37.956673 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:37.956642    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:37.957259 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:37.957089    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:37.958042 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:37.958023    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 12:40:59.982808 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:40:59.982679    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"]
Nov 01 12:41:00.046293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.036379    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.486312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.476503    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.486312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.476566    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.486312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.476613    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488973be-421e-4a45-929c-6538515e36b3-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.486312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.476672    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.486312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.476704    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488973be-421e-4a45-929c-6538515e36b3-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.486312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.476738    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.486312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.476828    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.486312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.476919    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.486312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.476983    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkm8\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-kube-api-access-wxkm8\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.667073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.666225    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.667073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.666389    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.667073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.666461    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkm8\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-kube-api-access-wxkm8\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.667073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.666633    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.667073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.666693    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.667073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.666733    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488973be-421e-4a45-929c-6538515e36b3-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.667073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.666801    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.667073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.666842    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488973be-421e-4a45-929c-6538515e36b3-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.667073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.666874    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.700080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.697868    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.700080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.699385    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.700080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.699865    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.700080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.700077    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488973be-421e-4a45-929c-6538515e36b3-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.700459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.700159    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.724251 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.724200    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.748194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.748105    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"]
Nov 01 12:41:00.759006 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.758941    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488973be-421e-4a45-929c-6538515e36b3-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.772402 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.771920    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkm8\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-kube-api-access-wxkm8\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.773638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.773603    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:00.943181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:00.943137    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:41:01.042398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:01.042308    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:01.106949 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:41:01.106911    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:41:01.618009 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:01.617974    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"]
Nov 01 12:41:01.624951 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:41:01.624916    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488973be_421e_4a45_929c_6538515e36b3.slice/crio-1230264d9527dd7d364f2e6e5fab6e4181d347e536a7b064444614602cb56676 WatchSource:0}: Error finding container 1230264d9527dd7d364f2e6e5fab6e4181d347e536a7b064444614602cb56676: Status 404 returned error can't find the container with id 1230264d9527dd7d364f2e6e5fab6e4181d347e536a7b064444614602cb56676
Nov 01 12:41:01.628317 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:01.628264    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:01.628445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:01.628369    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:01.628445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:01.628419    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:01.683901 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:01.683822    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" event={"ID":"488973be-421e-4a45-929c-6538515e36b3","Type":"ContainerStarted","Data":"1230264d9527dd7d364f2e6e5fab6e4181d347e536a7b064444614602cb56676"}
Nov 01 12:41:02.696127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:02.695740    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" event={"ID":"488973be-421e-4a45-929c-6538515e36b3","Type":"ContainerStarted","Data":"67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f"}
Nov 01 12:41:02.910765 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:02.910712    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" podStartSLOduration=3.910694827 podStartE2EDuration="3.910694827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:02.858611953 +0000 UTC m=+1104.365803171" watchObservedRunningTime="2026-11-01 12:41:02.910694827 +0000 UTC m=+1104.417886116"
Nov 01 12:41:03.051074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:03.045005    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:03.051074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:03.045171    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 12:41:03.051074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:03.045227    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 12:41:04.049148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:04.045509    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 12:41:04.049148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:04.045575    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 12:41:05.054087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:05.049506    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 12:41:05.054087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:05.049574    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 12:41:06.044585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:06.044547    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 12:41:06.044869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:06.044841    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 12:41:07.044073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:07.043445    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 12:41:07.044073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:07.043512    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 12:41:08.043392 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:08.043354    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 12:41:08.043610 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:08.043416    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 12:41:09.048330 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:09.048292    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:09.048881 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:09.048788    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:09.049611 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:09.049593    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:41:10.927136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:10.927100    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"]
Nov 01 12:41:10.932813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:10.932792    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:10.982562 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:10.982527    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"]
Nov 01 12:41:11.021223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.021185    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tmp-dir\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.021402 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.021235    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-dshm\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.021402 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.021367    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-home\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.021475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.021405    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-model-cache\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.021475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.021435    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjqc\" (UniqueName: \"kubernetes.io/projected/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-kube-api-access-qcjqc\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.021599 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.021577    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tls-certs\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.024520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.024493    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9514b0\"/\"distinct-w5-9514b0-kserve-self-signed-certs\""
Nov 01 12:41:11.125935 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.125893    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tls-certs\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.126180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.125997    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tmp-dir\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.126180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.126025    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-dshm\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.126180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.126097    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-home\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.126180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.126119    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-model-cache\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.126180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.126143    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjqc\" (UniqueName: \"kubernetes.io/projected/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-kube-api-access-qcjqc\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.127244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.127208    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tmp-dir\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.127402 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.127312    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-home\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.127558 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.127534    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-model-cache\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.129822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.129794    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tls-certs\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.138369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.138340    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-dshm\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.180796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.180712    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjqc\" (UniqueName: \"kubernetes.io/projected/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-kube-api-access-qcjqc\") pod \"distinct-w5-9514b0-kserve-789b75cb8f-j4xrh\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") " pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.247445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.247408    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:11.714094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.714059    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"]
Nov 01 12:41:11.719993 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:41:11.719961    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3eec3d7_74ed_42cf_9b7f_197ba51daf8d.slice/crio-f2cfb583388905c3bfef83a58faed403aadf47f70bf46aa62a0e0e4499d6ee9c WatchSource:0}: Error finding container f2cfb583388905c3bfef83a58faed403aadf47f70bf46aa62a0e0e4499d6ee9c: Status 404 returned error can't find the container with id f2cfb583388905c3bfef83a58faed403aadf47f70bf46aa62a0e0e4499d6ee9c
Nov 01 12:41:11.818149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.818089    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" event={"ID":"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d","Type":"ContainerStarted","Data":"aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7"}
Nov 01 12:41:11.818149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.818149    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" event={"ID":"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d","Type":"ContainerStarted","Data":"f2cfb583388905c3bfef83a58faed403aadf47f70bf46aa62a0e0e4499d6ee9c"}
Nov 01 12:41:11.927314 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:11.927257    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podStartSLOduration=1.92724181 podStartE2EDuration="1.92724181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:11.923898925 +0000 UTC m=+1113.431090149" watchObservedRunningTime="2026-11-01 12:41:11.92724181 +0000 UTC m=+1113.434433026"
Nov 01 12:41:12.247907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:12.247727    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:15.078433 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.078388    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"]
Nov 01 12:41:15.083020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.082990    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.157281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.157245    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"]
Nov 01 12:41:15.161694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.161670    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.184830 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.184784    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.185001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.184863    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5k4\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-kube-api-access-gp5k4\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.185001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.184965    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.185164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.185003    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.185164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.185045    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.185164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.185120    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.185323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.185166    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.185323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.185250    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.185323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.185298    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.188930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.188900    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"]
Nov 01 12:41:15.216987 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.216954    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"]
Nov 01 12:41:15.287492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.287462    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5k4\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-kube-api-access-gp5k4\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.287659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.287534    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.287659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.287562    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.287659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.287589    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.287659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.287620    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.287912 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.287674    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.287912 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.287862    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a449e-1000-4e98-84f3-c06545318c5c-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.287912 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.287904    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.288131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.287941    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.288131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288098    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.288238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288142    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.288238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288172    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.288238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288220    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.288399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288328    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.288399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288333    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp2jz\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-kube-api-access-cp2jz\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.288399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288391    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.288557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288415    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.288557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288447    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.288557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288487    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.288725 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288566    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.288803 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288781    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.288901 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.288880    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.289576 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.289549    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.290240 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.290212    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.290319 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.290298    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.341483 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.341415    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.341631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.341550    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5k4\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-kube-api-access-gp5k4\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.389449 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.389406    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.389612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.389474    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.389612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.389508    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a449e-1000-4e98-84f3-c06545318c5c-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.389612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.389546    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.389612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.389586    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.389825 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.389613    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.389825 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.389650    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.389825 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.389690    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp2jz\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-kube-api-access-cp2jz\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.389825 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.389757    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.390112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.390085    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.390210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.390118    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.390270 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.390220    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.390333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.390306    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a449e-1000-4e98-84f3-c06545318c5c-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.390524 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.390500    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.393599 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.393581    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.393838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.393820    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.399264 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.399239    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:15.445207 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.445177    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.445500 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.445471    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp2jz\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-kube-api-access-cp2jz\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.493517 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.493480    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:15.850266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.850237    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"]
Nov 01 12:41:15.895977 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:41:15.895943    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f43f391_f98e_45ff_8fdc_ea887e3b5115.slice/crio-f07f3ba9176a0e07f2860de75e54532fc3800de73e3d52774e5d9a26bcd535e5 WatchSource:0}: Error finding container f07f3ba9176a0e07f2860de75e54532fc3800de73e3d52774e5d9a26bcd535e5: Status 404 returned error can't find the container with id f07f3ba9176a0e07f2860de75e54532fc3800de73e3d52774e5d9a26bcd535e5
Nov 01 12:41:15.898350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.898315    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:15.898425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.898384    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:15.898425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.898420    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:15.904546 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.904519    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"]
Nov 01 12:41:15.909704 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:41:15.909679    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3a449e_1000_4e98_84f3_c06545318c5c.slice/crio-1f397058d0574a7e3322e06b93d4f6d31db9a7268963daf9538ba6ecce3d26fd WatchSource:0}: Error finding container 1f397058d0574a7e3322e06b93d4f6d31db9a7268963daf9538ba6ecce3d26fd: Status 404 returned error can't find the container with id 1f397058d0574a7e3322e06b93d4f6d31db9a7268963daf9538ba6ecce3d26fd
Nov 01 12:41:15.911826 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.911792    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:15.911902 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.911862    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:15.911945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:15.911906    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:16.893126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:16.893084    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" event={"ID":"8a3a449e-1000-4e98-84f3-c06545318c5c","Type":"ContainerStarted","Data":"1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf"}
Nov 01 12:41:16.893581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:16.893158    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" event={"ID":"8a3a449e-1000-4e98-84f3-c06545318c5c","Type":"ContainerStarted","Data":"1f397058d0574a7e3322e06b93d4f6d31db9a7268963daf9538ba6ecce3d26fd"}
Nov 01 12:41:16.895226 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:16.895203    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" event={"ID":"8f43f391-f98e-45ff-8fdc-ea887e3b5115","Type":"ContainerStarted","Data":"cf829ff5ba9ba77dc13e64f54e3b4528f403d78c4711dbc999e51dc13a2b9e29"}
Nov 01 12:41:16.895355 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:16.895241    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" event={"ID":"8f43f391-f98e-45ff-8fdc-ea887e3b5115","Type":"ContainerStarted","Data":"f07f3ba9176a0e07f2860de75e54532fc3800de73e3d52774e5d9a26bcd535e5"}
Nov 01 12:41:17.091130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:17.091029    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podStartSLOduration=2.091009093 podStartE2EDuration="2.091009093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:17.06351642 +0000 UTC m=+1118.570707622" watchObservedRunningTime="2026-11-01 12:41:17.091009093 +0000 UTC m=+1118.598200311"
Nov 01 12:41:17.189506 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:17.189028    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podStartSLOduration=2.189014641 podStartE2EDuration="2.189014641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:17.184553383 +0000 UTC m=+1118.691744601" watchObservedRunningTime="2026-11-01 12:41:17.189014641 +0000 UTC m=+1118.696205921"
Nov 01 12:41:17.399884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:17.399843    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:17.401455 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:17.401424    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 12:41:17.401605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:17.401479    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 12:41:17.495168 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:17.495122    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:17.496969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:17.496939    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 12:41:17.497172 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:17.496985    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 12:41:18.401506 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:18.401465    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 12:41:18.401881 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:18.401524    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 12:41:18.494800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:18.494757    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 12:41:18.495026 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:18.495003    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 12:41:19.400932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:19.400575    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 12:41:19.400932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:19.400631    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 12:41:19.495146 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:19.494791    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 12:41:19.495146 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:19.494850    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 12:41:20.399756 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:20.399715    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 12:41:20.399983 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:20.399782    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 12:41:20.495164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:20.495128    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 12:41:20.495656 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:20.495187    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 12:41:21.257075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:21.253486    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:21.300496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:21.300463    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:21.399766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:21.399731    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 12:41:21.399924 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:21.399787    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 12:41:21.495069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:21.495011    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 12:41:21.495223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:21.495088    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 12:41:21.992533 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:21.992420    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:41:22.400970 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:22.399761    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 12:41:22.400970 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:22.399836    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 12:41:22.495260 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:22.495220    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 12:41:22.495668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:22.495312    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 12:41:23.404151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:23.404119    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:23.404698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:23.404680    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:23.405443 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:23.405421    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:41:23.499812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:23.499777    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:23.500227 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:23.500167    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:23.501008 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:23.500989    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:41:31.039683 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:41:31.039644    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:41:45.164546 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.164510    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"]
Nov 01 12:41:45.170358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.170334    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.219631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.219596    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ecd640-openshift-default-dockercfg-mfhth\""
Nov 01 12:41:45.276609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.276572    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"]
Nov 01 12:41:45.317332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.316838    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-data\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.317797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.317775    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-envoy\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.317958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.317937    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04995d67-9144-429c-b655-0c1fd37ad11c-istiod-ca-cert\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.318093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.318044    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04995d67-9144-429c-b655-0c1fd37ad11c-istio-podinfo\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.318215 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.318187    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-credential-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.318397 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.318370    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-istio-token\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.318486 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.318409    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.318568 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.318550    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-certs\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.318647 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.318595    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdpqr\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-kube-api-access-zdpqr\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.322182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.322161    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"]
Nov 01 12:41:45.326683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.326659    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.376931 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.376897    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"]
Nov 01 12:41:45.420281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420186    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.420281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420236    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.420509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420328    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.420509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420411    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.420509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420490    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-certs\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.420650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420541    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdpqr\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-kube-api-access-zdpqr\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.420650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420573    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.420650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420637    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.420800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420666    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-data\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.420800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420692    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq6sv\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-kube-api-access-cq6sv\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.420800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420761    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-envoy\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.420800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420788    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04995d67-9144-429c-b655-0c1fd37ad11c-istiod-ca-cert\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.421011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420820    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.421011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420915    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04995d67-9144-429c-b655-0c1fd37ad11c-istio-podinfo\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.421011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420939    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.421011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420973    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-certs\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.421011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.420999    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-credential-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.421311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.421027    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-istio-token\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.421311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.421070    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.421311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.421187    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-data\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.421473 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.421430    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.421679 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.421657    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04995d67-9144-429c-b655-0c1fd37ad11c-istiod-ca-cert\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.422125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.422101    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-credential-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.424308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.424282    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-envoy\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.424424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.424407    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04995d67-9144-429c-b655-0c1fd37ad11c-istio-podinfo\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.474147 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.474103    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdpqr\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-kube-api-access-zdpqr\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.474290 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.474148    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-istio-token\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.485016 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.484992    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:45.521961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.521924    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq6sv\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-kube-api-access-cq6sv\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.522127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.522027    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.522209 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.522138    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.522269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.522239    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.522330 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.522266    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.522330 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.522294    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.522441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.522354    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.522441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.522405    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.522545 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.522516    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.525464 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.523118    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.525464 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.524639    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.525464 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.524893    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.525464 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.525149    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.525464 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.525426    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.525805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.525670    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.531284 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.531260    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.577432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.577370    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.578907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.578851    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq6sv\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-kube-api-access-cq6sv\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.641180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.641137    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:45.946111 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:41:45.942152    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04995d67_9144_429c_b655_0c1fd37ad11c.slice/crio-6a7656e9114535d1f300cde6ac1fe987d58dc7d315595e4e69b55e5ebb52cf68 WatchSource:0}: Error finding container 6a7656e9114535d1f300cde6ac1fe987d58dc7d315595e4e69b55e5ebb52cf68: Status 404 returned error can't find the container with id 6a7656e9114535d1f300cde6ac1fe987d58dc7d315595e4e69b55e5ebb52cf68
Nov 01 12:41:45.950426 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.950134    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:45.950426 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.950221    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:45.950426 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.950305    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:45.950912 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:45.950728    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"]
Nov 01 12:41:46.040277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.040245    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"]
Nov 01 12:41:46.043385 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:41:46.043350    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db77c69_a8d9_4a5a_8b1b_955a6b534748.slice/crio-1f41ff60349e360c68596ba5668f6db4ae91b8d2bd447f9ad7e2876ee87e29ed WatchSource:0}: Error finding container 1f41ff60349e360c68596ba5668f6db4ae91b8d2bd447f9ad7e2876ee87e29ed: Status 404 returned error can't find the container with id 1f41ff60349e360c68596ba5668f6db4ae91b8d2bd447f9ad7e2876ee87e29ed
Nov 01 12:41:46.046741 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.046702    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:46.046832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.046789    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:46.046877 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.046833    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:41:46.290711 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.290674    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" event={"ID":"04995d67-9144-429c-b655-0c1fd37ad11c","Type":"ContainerStarted","Data":"5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a"}
Nov 01 12:41:46.291250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.290734    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" event={"ID":"04995d67-9144-429c-b655-0c1fd37ad11c","Type":"ContainerStarted","Data":"6a7656e9114535d1f300cde6ac1fe987d58dc7d315595e4e69b55e5ebb52cf68"}
Nov 01 12:41:46.292843 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.292817    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" event={"ID":"6db77c69-a8d9-4a5a-8b1b-955a6b534748","Type":"ContainerStarted","Data":"4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d"}
Nov 01 12:41:46.292963 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.292865    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" event={"ID":"6db77c69-a8d9-4a5a-8b1b-955a6b534748","Type":"ContainerStarted","Data":"1f41ff60349e360c68596ba5668f6db4ae91b8d2bd447f9ad7e2876ee87e29ed"}
Nov 01 12:41:46.407869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.407788    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" podStartSLOduration=1.407759763 podStartE2EDuration="1.407759763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:46.403818943 +0000 UTC m=+1147.911010160" watchObservedRunningTime="2026-11-01 12:41:46.407759763 +0000 UTC m=+1147.914950984"
Nov 01 12:41:46.485711 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.485678    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:46.522722 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.522658    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podStartSLOduration=1.522636605 podStartE2EDuration="1.522636605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:46.514497475 +0000 UTC m=+1148.021688704" watchObservedRunningTime="2026-11-01 12:41:46.522636605 +0000 UTC m=+1148.029827823"
Nov 01 12:41:46.651126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:46.647314    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:47.486781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:47.486744    5725 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 12:41:47.487179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:47.486801    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 12:41:47.651076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:47.649291    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 12:41:47.651076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:47.649355    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 12:41:48.486207 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:48.485773    5725 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 12:41:48.486207 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:48.485837    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 12:41:48.643077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:48.642591    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 12:41:48.643077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:48.642653    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 12:41:49.486195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:49.485698    5725 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 12:41:49.486195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:49.485766    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 12:41:49.645491 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:49.645451    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 12:41:49.646106 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:49.646042    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 12:41:50.486073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:50.485569    5725 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 12:41:50.486073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:50.485646    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 12:41:50.645508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:50.645467    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 12:41:50.646157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:50.646126    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 12:41:51.488496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:51.488457    5725 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 12:41:51.489449 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:51.489392    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 12:41:51.652073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:51.649849    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 12:41:51.652073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:51.649911    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 12:41:52.485975 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:52.485937    5725 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 12:41:52.486173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:52.485999    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 12:41:52.641938 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:52.641889    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 12:41:52.642176 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:52.641964    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 12:41:53.490279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:53.490249    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:53.490813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:53.490786    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:53.491516 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:53.491493    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:41:53.648186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:53.648151    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:53.648522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:53.648498    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:41:53.649328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:41:53.649301    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:42:01.028949 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:42:01.028913    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:42:06.482061 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.481996    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"]
Nov 01 12:42:06.491092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.490503    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.533282 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.533245    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"]
Nov 01 12:42:06.576440 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.576404    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:42:06.577733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.577702    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:42:06.586127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.586089    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1064198f-wmmfs\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.586127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.586128    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbn8\" (UniqueName: \"kubernetes.io/projected/12363f69-ecc4-49c3-9633-94b5c3f01ae2-kube-api-access-8bbn8\") pod \"maas-api-revoke-sub-1064198f-wmmfs\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.586308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.586155    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1064198f-wmmfs\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.688074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.687612    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1064198f-wmmfs\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.688074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.687664    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbn8\" (UniqueName: \"kubernetes.io/projected/12363f69-ecc4-49c3-9633-94b5c3f01ae2-kube-api-access-8bbn8\") pod \"maas-api-revoke-sub-1064198f-wmmfs\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.688074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.687708    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1064198f-wmmfs\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.693082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.688863    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1064198f-wmmfs\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.703877 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.703799    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1064198f-wmmfs\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.743360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.743285    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbn8\" (UniqueName: \"kubernetes.io/projected/12363f69-ecc4-49c3-9633-94b5c3f01ae2-kube-api-access-8bbn8\") pod \"maas-api-revoke-sub-1064198f-wmmfs\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:06.806720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:06.806675    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:07.412569 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:07.412537    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"]
Nov 01 12:42:07.417532 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:42:07.417503    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12363f69_ecc4_49c3_9633_94b5c3f01ae2.slice/crio-f3b63e9254c8655a8bc9f81f88bcedefd6ff06442de3fca78f39c334ef4fe231 WatchSource:0}: Error finding container f3b63e9254c8655a8bc9f81f88bcedefd6ff06442de3fca78f39c334ef4fe231: Status 404 returned error can't find the container with id f3b63e9254c8655a8bc9f81f88bcedefd6ff06442de3fca78f39c334ef4fe231
Nov 01 12:42:07.498707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:07.498673    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs" event={"ID":"12363f69-ecc4-49c3-9633-94b5c3f01ae2","Type":"ContainerStarted","Data":"f3b63e9254c8655a8bc9f81f88bcedefd6ff06442de3fca78f39c334ef4fe231"}
Nov 01 12:42:08.508521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:08.508489    5725 generic.go:356] "Generic (PLEG): container finished" podID="12363f69-ecc4-49c3-9633-94b5c3f01ae2" containerID="e4531fe4e420b961ff3f64f1ba44b2445ebb37c9f2acd58ffd415b39a4092595" exitCode=0
Nov 01 12:42:08.508876 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:08.508588    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs" event={"ID":"12363f69-ecc4-49c3-9633-94b5c3f01ae2","Type":"ContainerDied","Data":"e4531fe4e420b961ff3f64f1ba44b2445ebb37c9f2acd58ffd415b39a4092595"}
Nov 01 12:42:09.517999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.517967    5725 generic.go:356] "Generic (PLEG): container finished" podID="3c751f45-9cb5-4c77-b4c9-1c30ea327ff8" containerID="527efbd7c1bcb39f00b1fa9a30c536cbe2c5a4cbfba6e932ea3b813776017d03" exitCode=0
Nov 01 12:42:09.518429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.518037    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rnj8v-hj8cd" event={"ID":"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8","Type":"ContainerDied","Data":"527efbd7c1bcb39f00b1fa9a30c536cbe2c5a4cbfba6e932ea3b813776017d03"}
Nov 01 12:42:09.689729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.689701    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:09.741868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.741836    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-xks-ca\") pod \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") "
Nov 01 12:42:09.742074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.741975    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12363f69-ecc4-49c3-9633-94b5c3f01ae2-kube-api-access-8bbn8\" (UniqueName: \"kubernetes.io/projected/12363f69-ecc4-49c3-9633-94b5c3f01ae2-kube-api-access-8bbn8\") pod \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") "
Nov 01 12:42:09.742162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.742072    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-openshift-ca\") pod \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\" (UID: \"12363f69-ecc4-49c3-9633-94b5c3f01ae2\") "
Nov 01 12:42:09.742472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.742445    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-openshift-ca" pod "12363f69-ecc4-49c3-9633-94b5c3f01ae2" (UID: "12363f69-ecc4-49c3-9633-94b5c3f01ae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:42:09.742630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.742613    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:09.744195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.744169    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-xks-ca" pod "12363f69-ecc4-49c3-9633-94b5c3f01ae2" (UID: "12363f69-ecc4-49c3-9633-94b5c3f01ae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:42:09.744609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.744590    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12363f69-ecc4-49c3-9633-94b5c3f01ae2-kube-api-access-8bbn8" pod "12363f69-ecc4-49c3-9633-94b5c3f01ae2" (UID: "12363f69-ecc4-49c3-9633-94b5c3f01ae2"). InnerVolumeSpecName "kube-api-access-8bbn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:42:09.856930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.847676    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbn8\" (UniqueName: \"kubernetes.io/projected/12363f69-ecc4-49c3-9633-94b5c3f01ae2-kube-api-access-8bbn8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:09.856930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:09.847716    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12363f69-ecc4-49c3-9633-94b5c3f01ae2-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:10.528668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.528639    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"
Nov 01 12:42:10.529157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.528660    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs" event={"ID":"12363f69-ecc4-49c3-9633-94b5c3f01ae2","Type":"ContainerDied","Data":"f3b63e9254c8655a8bc9f81f88bcedefd6ff06442de3fca78f39c334ef4fe231"}
Nov 01 12:42:10.529157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.528752    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b63e9254c8655a8bc9f81f88bcedefd6ff06442de3fca78f39c334ef4fe231"
Nov 01 12:42:10.708453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.708425    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:42:10.758497 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.758466    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-service-ca-bundle\") pod \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") "
Nov 01 12:42:10.758670 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.758531    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-archives-path\") pod \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") "
Nov 01 12:42:10.758670 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.758594    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-trusted-ca-bundle\") pod \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") "
Nov 01 12:42:10.758796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.758783    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-kube-api-access-6pk2r\" (UniqueName: \"kubernetes.io/projected/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-kube-api-access-6pk2r\") pod \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\" (UID: \"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8\") "
Nov 01 12:42:10.758889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.758866    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-service-ca-bundle" pod "3c751f45-9cb5-4c77-b4c9-1c30ea327ff8" (UID: "3c751f45-9cb5-4c77-b4c9-1c30ea327ff8"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:42:10.758931 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.758900    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-archives-path" pod "3c751f45-9cb5-4c77-b4c9-1c30ea327ff8" (UID: "3c751f45-9cb5-4c77-b4c9-1c30ea327ff8"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:42:10.758989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.758972    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-trusted-ca-bundle" pod "3c751f45-9cb5-4c77-b4c9-1c30ea327ff8" (UID: "3c751f45-9cb5-4c77-b4c9-1c30ea327ff8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:42:10.759406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.759387    5725 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-trusted-ca-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:10.759406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.759406    5725 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-service-ca-bundle\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:10.759555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.759415    5725 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-archives-path\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:10.761011 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.760989    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-kube-api-access-6pk2r" pod "3c751f45-9cb5-4c77-b4c9-1c30ea327ff8" (UID: "3c751f45-9cb5-4c77-b4c9-1c30ea327ff8"). InnerVolumeSpecName "kube-api-access-6pk2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:42:10.860701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:10.860626    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pk2r\" (UniqueName: \"kubernetes.io/projected/3c751f45-9cb5-4c77-b4c9-1c30ea327ff8-kube-api-access-6pk2r\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:11.537746 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:11.537712    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rnj8v-hj8cd"
Nov 01 12:42:11.538163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:11.537745    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rnj8v-hj8cd" event={"ID":"3c751f45-9cb5-4c77-b4c9-1c30ea327ff8","Type":"ContainerDied","Data":"da0e221f666a47a91f0a708a5d2f26566a933f72cede0f214f7c37a135ed8da8"}
Nov 01 12:42:11.538163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:11.537794    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da0e221f666a47a91f0a708a5d2f26566a933f72cede0f214f7c37a135ed8da8"
Nov 01 12:42:31.028717 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:42:31.028635    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:42:34.692813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.692768    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"]
Nov 01 12:42:34.693661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.693639    5725 cpu_manager.go:414] "containerMap: removing container" podUID="12363f69-ecc4-49c3-9633-94b5c3f01ae2" containerName="revoke-keys"
Nov 01 12:42:34.693767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.693662    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12363f69-ecc4-49c3-9633-94b5c3f01ae2" containerName="revoke-keys"
Nov 01 12:42:34.693767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.693685    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3c751f45-9cb5-4c77-b4c9-1c30ea327ff8" containerName="insights-gathering"
Nov 01 12:42:34.693767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.693693    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c751f45-9cb5-4c77-b4c9-1c30ea327ff8" containerName="insights-gathering"
Nov 01 12:42:34.693933 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.693830    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="12363f69-ecc4-49c3-9633-94b5c3f01ae2" containerName="revoke-keys"
Nov 01 12:42:34.693933 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.693845    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c751f45-9cb5-4c77-b4c9-1c30ea327ff8" containerName="insights-gathering"
Nov 01 12:42:34.697581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.697557    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:34.749156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.749124    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"]
Nov 01 12:42:34.789373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.789334    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nzvl\" (UniqueName: \"kubernetes.io/projected/22225b05-f5e3-4911-9240-38c7431eb4a9-kube-api-access-4nzvl\") pod \"maas-api-revoke-sub-8fb5da01-hj5sb\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:34.789546 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.789405    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb5da01-hj5sb\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:34.789659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.789606    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb5da01-hj5sb\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:34.790263 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.790237    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:42:34.790423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.790337    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:42:34.890729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.890693    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nzvl\" (UniqueName: \"kubernetes.io/projected/22225b05-f5e3-4911-9240-38c7431eb4a9-kube-api-access-4nzvl\") pod \"maas-api-revoke-sub-8fb5da01-hj5sb\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:34.890930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.890746    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb5da01-hj5sb\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:34.890930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.890824    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb5da01-hj5sb\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:34.891593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.891571    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb5da01-hj5sb\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:34.893254 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.893233    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb5da01-hj5sb\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:34.944608 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:34.944538    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nzvl\" (UniqueName: \"kubernetes.io/projected/22225b05-f5e3-4911-9240-38c7431eb4a9-kube-api-access-4nzvl\") pod \"maas-api-revoke-sub-8fb5da01-hj5sb\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:35.011547 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:35.011510    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:35.392871 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:35.392836    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"]
Nov 01 12:42:35.398324 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:42:35.398287    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22225b05_f5e3_4911_9240_38c7431eb4a9.slice/crio-e357b91dd59f119c6377ce1882290de45414b977beae52362c3d180850bd381c WatchSource:0}: Error finding container e357b91dd59f119c6377ce1882290de45414b977beae52362c3d180850bd381c: Status 404 returned error can't find the container with id e357b91dd59f119c6377ce1882290de45414b977beae52362c3d180850bd381c
Nov 01 12:42:35.750978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:35.750944    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb" event={"ID":"22225b05-f5e3-4911-9240-38c7431eb4a9","Type":"ContainerStarted","Data":"323b404755ba397de73a453fa62c035c3edc27806824ed0813624a9848f1735d"}
Nov 01 12:42:35.751375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:35.751004    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb" event={"ID":"22225b05-f5e3-4911-9240-38c7431eb4a9","Type":"ContainerStarted","Data":"e357b91dd59f119c6377ce1882290de45414b977beae52362c3d180850bd381c"}
Nov 01 12:42:35.858028 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:35.857979    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb" podStartSLOduration=1.857963116 podStartE2EDuration="1.857963116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:35.857193067 +0000 UTC m=+1197.364384297" watchObservedRunningTime="2026-11-01 12:42:35.857963116 +0000 UTC m=+1197.365154332"
Nov 01 12:42:36.761210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:36.761168    5725 generic.go:356] "Generic (PLEG): container finished" podID="22225b05-f5e3-4911-9240-38c7431eb4a9" containerID="323b404755ba397de73a453fa62c035c3edc27806824ed0813624a9848f1735d" exitCode=0
Nov 01 12:42:36.761706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:36.761276    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb" event={"ID":"22225b05-f5e3-4911-9240-38c7431eb4a9","Type":"ContainerDied","Data":"323b404755ba397de73a453fa62c035c3edc27806824ed0813624a9848f1735d"}
Nov 01 12:42:37.953321 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:37.953295    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:38.029411 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.029320    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-xks-ca\") pod \"22225b05-f5e3-4911-9240-38c7431eb4a9\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") "
Nov 01 12:42:38.029411 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.029396    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22225b05-f5e3-4911-9240-38c7431eb4a9-kube-api-access-4nzvl\" (UniqueName: \"kubernetes.io/projected/22225b05-f5e3-4911-9240-38c7431eb4a9-kube-api-access-4nzvl\") pod \"22225b05-f5e3-4911-9240-38c7431eb4a9\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") "
Nov 01 12:42:38.029810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.029527    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-openshift-ca\") pod \"22225b05-f5e3-4911-9240-38c7431eb4a9\" (UID: \"22225b05-f5e3-4911-9240-38c7431eb4a9\") "
Nov 01 12:42:38.029923 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.029893    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-openshift-ca" pod "22225b05-f5e3-4911-9240-38c7431eb4a9" (UID: "22225b05-f5e3-4911-9240-38c7431eb4a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:42:38.030253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.030230    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:38.031673 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.031643    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22225b05-f5e3-4911-9240-38c7431eb4a9-kube-api-access-4nzvl" pod "22225b05-f5e3-4911-9240-38c7431eb4a9" (UID: "22225b05-f5e3-4911-9240-38c7431eb4a9"). InnerVolumeSpecName "kube-api-access-4nzvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:42:38.031935 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.031912    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-xks-ca" pod "22225b05-f5e3-4911-9240-38c7431eb4a9" (UID: "22225b05-f5e3-4911-9240-38c7431eb4a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:42:38.131390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.131349    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22225b05-f5e3-4911-9240-38c7431eb4a9-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:38.131390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.131385    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nzvl\" (UniqueName: \"kubernetes.io/projected/22225b05-f5e3-4911-9240-38c7431eb4a9-kube-api-access-4nzvl\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:38.780811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.780774    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"
Nov 01 12:42:38.780811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.780795    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb" event={"ID":"22225b05-f5e3-4911-9240-38c7431eb4a9","Type":"ContainerDied","Data":"e357b91dd59f119c6377ce1882290de45414b977beae52362c3d180850bd381c"}
Nov 01 12:42:38.781023 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.780852    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e357b91dd59f119c6377ce1882290de45414b977beae52362c3d180850bd381c"
Nov 01 12:42:38.904189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.904110    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:42:38.905694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.905670    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:42:38.911945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.911925    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:42:38.913242 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:38.913226    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:42:45.738108 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.738064    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"]
Nov 01 12:42:45.738999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.738973    5725 cpu_manager.go:414] "containerMap: removing container" podUID="22225b05-f5e3-4911-9240-38c7431eb4a9" containerName="revoke-keys"
Nov 01 12:42:45.738999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.738999    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22225b05-f5e3-4911-9240-38c7431eb4a9" containerName="revoke-keys"
Nov 01 12:42:45.739209 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.739199    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="22225b05-f5e3-4911-9240-38c7431eb4a9" containerName="revoke-keys"
Nov 01 12:42:45.744825 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.744802    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.846287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.841703    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.846287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.841770    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.846287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.841868    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.846287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.841911    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd0b429-5570-493d-8de7-9a315bae15c2-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.846287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.841943    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v64fb\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-kube-api-access-v64fb\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.846287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.842102    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.846287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.842180    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.846287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.842241    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.846287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.842285    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.935758    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"]
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944006    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944163    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944217    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944248    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944326    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944357    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944427    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944474    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd0b429-5570-493d-8de7-9a315bae15c2-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944481    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944498    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v64fb\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-kube-api-access-v64fb\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.944907    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.945159    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.945663    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd0b429-5570-493d-8de7-9a315bae15c2-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.946988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.945787    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.948275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.948250    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.948679 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.948657    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.998363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.998284    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v64fb\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-kube-api-access-v64fb\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:45.998751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:45.998727    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:46.061903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:46.061867    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:46.506516 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:46.506480    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"]
Nov 01 12:42:46.512395 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:42:46.512363    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd0b429_5570_493d_8de7_9a315bae15c2.slice/crio-aded86bd8e8cc80761208a82a9c3f8100fc8cd3eb4b82ef4e770f2e0a0e1e22c WatchSource:0}: Error finding container aded86bd8e8cc80761208a82a9c3f8100fc8cd3eb4b82ef4e770f2e0a0e1e22c: Status 404 returned error can't find the container with id aded86bd8e8cc80761208a82a9c3f8100fc8cd3eb4b82ef4e770f2e0a0e1e22c
Nov 01 12:42:46.515288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:46.515251    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:42:46.515390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:46.515337    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:42:46.515390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:46.515381    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:42:46.855310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:46.855198    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" event={"ID":"3bd0b429-5570-493d-8de7-9a315bae15c2","Type":"ContainerStarted","Data":"60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298"}
Nov 01 12:42:46.855310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:46.855272    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" event={"ID":"3bd0b429-5570-493d-8de7-9a315bae15c2","Type":"ContainerStarted","Data":"aded86bd8e8cc80761208a82a9c3f8100fc8cd3eb4b82ef4e770f2e0a0e1e22c"}
Nov 01 12:42:46.970281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:46.970222    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podStartSLOduration=1.970204112 podStartE2EDuration="1.970204112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:46.966720898 +0000 UTC m=+1208.473912138" watchObservedRunningTime="2026-11-01 12:42:46.970204112 +0000 UTC m=+1208.477395337"
Nov 01 12:42:47.062820 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:47.062778    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:47.064595 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:47.064565    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 01 12:42:47.064754 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:47.064621    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 01 12:42:48.063485 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:48.063436    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 01 12:42:48.063905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:48.063497    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 01 12:42:49.065067 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:49.065015    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 01 12:42:49.065655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:49.065622    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 01 12:42:50.063422 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.063388    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 01 12:42:50.063743 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.063689    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 01 12:42:50.267086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.267035    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"]
Nov 01 12:42:50.272921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.272898    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.304579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.304545    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2235855b-5kbnx\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.306836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.305586    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"]
Nov 01 12:42:50.307070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.306911    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqszw\" (UniqueName: \"kubernetes.io/projected/5848c802-b889-40d6-a7e9-2cc485e01847-kube-api-access-jqszw\") pod \"maas-api-revoke-sub-2235855b-5kbnx\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.307282 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.307264    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2235855b-5kbnx\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.322537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.322446    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:42:50.350336 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.350264    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:42:50.409179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.409135    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2235855b-5kbnx\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.409358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.409316    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqszw\" (UniqueName: \"kubernetes.io/projected/5848c802-b889-40d6-a7e9-2cc485e01847-kube-api-access-jqszw\") pod \"maas-api-revoke-sub-2235855b-5kbnx\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.409403 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.409386    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2235855b-5kbnx\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.409975 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.409958    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2235855b-5kbnx\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.411826 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.411802    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2235855b-5kbnx\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.464639 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.464603    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqszw\" (UniqueName: \"kubernetes.io/projected/5848c802-b889-40d6-a7e9-2cc485e01847-kube-api-access-jqszw\") pod \"maas-api-revoke-sub-2235855b-5kbnx\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:50.589899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:50.589812    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:51.009553 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:51.009521    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"]
Nov 01 12:42:51.063882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:51.063068    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 01 12:42:51.063882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:51.063128    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 01 12:42:51.903704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:51.903670    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx" event={"ID":"5848c802-b889-40d6-a7e9-2cc485e01847","Type":"ContainerStarted","Data":"48448f3d49fc07de8445637ea0882863bea6515d6cb3aa6206c9f1e0135270b9"}
Nov 01 12:42:51.904150 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:51.903730    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx" event={"ID":"5848c802-b889-40d6-a7e9-2cc485e01847","Type":"ContainerStarted","Data":"634dcd61d59e0059eb865aef1c087fdc92c4bf77075c05a1a7dc3164c2126bd0"}
Nov 01 12:42:52.015069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:52.014993    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx" podStartSLOduration=2.014974052 podStartE2EDuration="2.014974052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:52.011505509 +0000 UTC m=+1213.518696726" watchObservedRunningTime="2026-11-01 12:42:52.014974052 +0000 UTC m=+1213.522165268"
Nov 01 12:42:52.062959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:52.062926    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 01 12:42:52.063167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:52.062993    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 01 12:42:52.913654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:52.913616    5725 generic.go:356] "Generic (PLEG): container finished" podID="5848c802-b889-40d6-a7e9-2cc485e01847" containerID="48448f3d49fc07de8445637ea0882863bea6515d6cb3aa6206c9f1e0135270b9" exitCode=0
Nov 01 12:42:52.914073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:52.913695    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx" event={"ID":"5848c802-b889-40d6-a7e9-2cc485e01847","Type":"ContainerDied","Data":"48448f3d49fc07de8445637ea0882863bea6515d6cb3aa6206c9f1e0135270b9"}
Nov 01 12:42:53.062903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:53.062864    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Nov 01 12:42:53.063155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:53.062924    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Nov 01 12:42:54.113091 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.113058    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:54.179432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.179335    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-xks-ca\") pod \"5848c802-b889-40d6-a7e9-2cc485e01847\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") "
Nov 01 12:42:54.179615 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.179592    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5848c802-b889-40d6-a7e9-2cc485e01847-kube-api-access-jqszw\" (UniqueName: \"kubernetes.io/projected/5848c802-b889-40d6-a7e9-2cc485e01847-kube-api-access-jqszw\") pod \"5848c802-b889-40d6-a7e9-2cc485e01847\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") "
Nov 01 12:42:54.179685 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.179641    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-openshift-ca\") pod \"5848c802-b889-40d6-a7e9-2cc485e01847\" (UID: \"5848c802-b889-40d6-a7e9-2cc485e01847\") "
Nov 01 12:42:54.180198 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.180150    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-openshift-ca" pod "5848c802-b889-40d6-a7e9-2cc485e01847" (UID: "5848c802-b889-40d6-a7e9-2cc485e01847"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:42:54.182179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.182145    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5848c802-b889-40d6-a7e9-2cc485e01847-kube-api-access-jqszw" pod "5848c802-b889-40d6-a7e9-2cc485e01847" (UID: "5848c802-b889-40d6-a7e9-2cc485e01847"). InnerVolumeSpecName "kube-api-access-jqszw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:42:54.182304 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.182284    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-xks-ca" pod "5848c802-b889-40d6-a7e9-2cc485e01847" (UID: "5848c802-b889-40d6-a7e9-2cc485e01847"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:42:54.281706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.281659    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:54.281706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.281697    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqszw\" (UniqueName: \"kubernetes.io/projected/5848c802-b889-40d6-a7e9-2cc485e01847-kube-api-access-jqszw\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:54.281963 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.281714    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5848c802-b889-40d6-a7e9-2cc485e01847-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:42:54.325764 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.325726    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:54.326237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.326175    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:54.327515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.327493    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:42:54.933593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.933501    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"
Nov 01 12:42:54.933593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.933535    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx" event={"ID":"5848c802-b889-40d6-a7e9-2cc485e01847","Type":"ContainerDied","Data":"634dcd61d59e0059eb865aef1c087fdc92c4bf77075c05a1a7dc3164c2126bd0"}
Nov 01 12:42:54.933593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:54.933588    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="634dcd61d59e0059eb865aef1c087fdc92c4bf77075c05a1a7dc3164c2126bd0"
Nov 01 12:42:56.021109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.021073    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"]
Nov 01 12:42:56.021858 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.021839    5725 cpu_manager.go:414] "containerMap: removing container" podUID="5848c802-b889-40d6-a7e9-2cc485e01847" containerName="revoke-keys"
Nov 01 12:42:56.021920 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.021860    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5848c802-b889-40d6-a7e9-2cc485e01847" containerName="revoke-keys"
Nov 01 12:42:56.021991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.021982    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="5848c802-b889-40d6-a7e9-2cc485e01847" containerName="revoke-keys"
Nov 01 12:42:56.025970 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.025949    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.107694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.107659    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-tmp-dir\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.107889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.107807    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77084dd3-4b8d-4314-925d-478db9af9706-tls-certs\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.107889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.107867    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-home\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.107986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.107935    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-dshm\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.108020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.107987    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-model-cache\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.108079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.108044    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86l9l\" (UniqueName: \"kubernetes.io/projected/77084dd3-4b8d-4314-925d-478db9af9706-kube-api-access-86l9l\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.124460 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.124420    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"distinct-2-w4-06fd6b-kserve-self-signed-certs\""
Nov 01 12:42:56.129738 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.129708    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"]
Nov 01 12:42:56.209423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.209362    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86l9l\" (UniqueName: \"kubernetes.io/projected/77084dd3-4b8d-4314-925d-478db9af9706-kube-api-access-86l9l\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.209629 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.209458    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-tmp-dir\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.209629 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.209549    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77084dd3-4b8d-4314-925d-478db9af9706-tls-certs\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.209759 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.209705    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-home\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.209818 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.209788    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-dshm\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.209871 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.209838    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-model-cache\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.210065 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.210032    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-tmp-dir\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.210183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.210122    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-home\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.210313 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.210291    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-model-cache\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.212099 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.212038    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-dshm\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.213431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.213408    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77084dd3-4b8d-4314-925d-478db9af9706-tls-certs\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.265025 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.264987    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86l9l\" (UniqueName: \"kubernetes.io/projected/77084dd3-4b8d-4314-925d-478db9af9706-kube-api-access-86l9l\") pod \"distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.341247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.341153    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:42:56.813390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.813340    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"]
Nov 01 12:42:56.819314 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:42:56.819288    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77084dd3_4b8d_4314_925d_478db9af9706.slice/crio-8810056f8b67dc46a6f0af0c6c726143644d4372c0b5b9cfbeda06ee5932499c WatchSource:0}: Error finding container 8810056f8b67dc46a6f0af0c6c726143644d4372c0b5b9cfbeda06ee5932499c: Status 404 returned error can't find the container with id 8810056f8b67dc46a6f0af0c6c726143644d4372c0b5b9cfbeda06ee5932499c
Nov 01 12:42:56.951921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.951885    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" event={"ID":"77084dd3-4b8d-4314-925d-478db9af9706","Type":"ContainerStarted","Data":"573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948"}
Nov 01 12:42:56.952099 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:56.951931    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" event={"ID":"77084dd3-4b8d-4314-925d-478db9af9706","Type":"ContainerStarted","Data":"8810056f8b67dc46a6f0af0c6c726143644d4372c0b5b9cfbeda06ee5932499c"}
Nov 01 12:42:57.065129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:57.061384    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podStartSLOduration=2.061364227 podStartE2EDuration="2.061364227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:57.059132087 +0000 UTC m=+1218.566323302" watchObservedRunningTime="2026-11-01 12:42:57.061364227 +0000 UTC m=+1218.568555445"
Nov 01 12:42:57.342112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:42:57.342009    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:43:00.698428 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.698379    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"]
Nov 01 12:43:00.702644 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.702623    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.745475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.745427    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"]
Nov 01 12:43:00.749856 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.749828    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.779536 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.779499    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.779724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.779603    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.779724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.779648    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da118069-a12c-4401-b3ce-279be31f094c-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.779847 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.779721    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.779847 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.779825    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.779948 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.779926    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da118069-a12c-4401-b3ce-279be31f094c-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.780044 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.780027    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.780124 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.780085    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.780178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.780122    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkcvl\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-kube-api-access-kkcvl\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.800202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.800163    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"]
Nov 01 12:43:00.862965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.862931    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"]
Nov 01 12:43:00.884201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884149    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.884403 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884240    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.884403 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884304    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.884403 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884351    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da118069-a12c-4401-b3ce-279be31f094c-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.884583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884404    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.884583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884470    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvxxh\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-kube-api-access-xvxxh\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.884583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884507    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.884583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884534    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.884583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884562    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.884831 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884601    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkcvl\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-kube-api-access-kkcvl\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.884831 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.884674    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.885340 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.885312    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.885457 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.885356    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.885457 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.885386    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.885457 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.885434    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.885457 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.885442    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.885676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.885480    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.885676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.885531    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da118069-a12c-4401-b3ce-279be31f094c-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.885676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.885575    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.885835 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.885775    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.886095 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.886072    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.886320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.886295    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.887261 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.887234    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da118069-a12c-4401-b3ce-279be31f094c-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.889291 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.888883    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.889291 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.889245    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da118069-a12c-4401-b3ce-279be31f094c-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.940100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.940063    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.954911 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.954813    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkcvl\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-kube-api-access-kkcvl\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:00.988287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.986511    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.988287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.986605    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.988287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.986634    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.988287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.986665    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.988287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.986777    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.988287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.986825    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.988287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.986891    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.988287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.986949    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvxxh\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-kube-api-access-xvxxh\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.988287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.986985    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.992068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.989305    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.992068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.989332    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.992068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.989600    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.992068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.989817    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.992401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.992361    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:00.995754 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:00.995723    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:01.007969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.007862    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:01.017758 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.017724    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:01.042039 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:43:01.041998    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:43:01.043038 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.043006    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvxxh\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-kube-api-access-xvxxh\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:01.043161 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.043006    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:01.068266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.068223    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:01.473620 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.473580    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"]
Nov 01 12:43:01.519847 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:43:01.519813    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda118069_a12c_4401_b3ce_279be31f094c.slice/crio-96f76a5dc4909801d59a1528780f7444abaf7a9620ca8820d353eddae513e87d WatchSource:0}: Error finding container 96f76a5dc4909801d59a1528780f7444abaf7a9620ca8820d353eddae513e87d: Status 404 returned error can't find the container with id 96f76a5dc4909801d59a1528780f7444abaf7a9620ca8820d353eddae513e87d
Nov 01 12:43:01.523327 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.523280    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:01.523418 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.523394    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:01.523461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.523437    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:01.526842 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.526817    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"]
Nov 01 12:43:01.549353 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:43:01.549316    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4447d89_8ce2_4fc8_88e5_8abad0d8bd4b.slice/crio-9b5384c39c39804f86790debd958a86e002279d7b6c93f3a603e478950d33fb6 WatchSource:0}: Error finding container 9b5384c39c39804f86790debd958a86e002279d7b6c93f3a603e478950d33fb6: Status 404 returned error can't find the container with id 9b5384c39c39804f86790debd958a86e002279d7b6c93f3a603e478950d33fb6
Nov 01 12:43:01.552121 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.552042    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:01.552229 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.552155    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:01.552229 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:01.552189    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:02.009245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.009192    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" event={"ID":"da118069-a12c-4401-b3ce-279be31f094c","Type":"ContainerStarted","Data":"bf64ed1286f90583b4af2f6fdbd9c9b3b88a1f9fa32a83e593ff8c4451b2edb8"}
Nov 01 12:43:02.009735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.009269    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" event={"ID":"da118069-a12c-4401-b3ce-279be31f094c","Type":"ContainerStarted","Data":"96f76a5dc4909801d59a1528780f7444abaf7a9620ca8820d353eddae513e87d"}
Nov 01 12:43:02.011320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.011288    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" event={"ID":"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b","Type":"ContainerStarted","Data":"4c3d067e319c03646741e69ed093009bb2f0e32da3cd8890d5a63f22a001cdb7"}
Nov 01 12:43:02.011465 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.011343    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" event={"ID":"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b","Type":"ContainerStarted","Data":"9b5384c39c39804f86790debd958a86e002279d7b6c93f3a603e478950d33fb6"}
Nov 01 12:43:02.018228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.018184    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:02.020810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.020777    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 12:43:02.020928 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.020860    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 12:43:02.069179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.069138    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:02.072633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.072595    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 12:43:02.072785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.072656    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 12:43:02.120659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.120596    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podStartSLOduration=2.120576789 podStartE2EDuration="2.120576789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:02.119067569 +0000 UTC m=+1223.626258778" watchObservedRunningTime="2026-11-01 12:43:02.120576789 +0000 UTC m=+1223.627768009"
Nov 01 12:43:02.235337 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:02.235273    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podStartSLOduration=2.235254421 podStartE2EDuration="2.235254421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:02.234490161 +0000 UTC m=+1223.741681382" watchObservedRunningTime="2026-11-01 12:43:02.235254421 +0000 UTC m=+1223.742445639"
Nov 01 12:43:03.058072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:03.045663    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 12:43:03.058072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:03.045727    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 12:43:03.079336 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:03.079206    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 12:43:03.079336 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:03.079273    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 12:43:04.018886 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:04.018849    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 12:43:04.019098 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:04.018916    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 12:43:04.083073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:04.082384    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"]
Nov 01 12:43:04.083073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:04.082798    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 12:43:04.083073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:04.082849    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 12:43:04.100433 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:04.100399    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea0a74ae-xxm8w"]
Nov 01 12:43:04.617696 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:04.617663    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2303129-8a80-4d6b-83cd-6ca548c8ca43" path="/var/lib/kubelet/pods/e2303129-8a80-4d6b-83cd-6ca548c8ca43/volumes"
Nov 01 12:43:05.018382 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:05.018349    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 12:43:05.018655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:05.018632    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 12:43:05.077075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:05.071515    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 12:43:05.077075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:05.071578    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 12:43:06.018805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:06.018765    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 12:43:06.019242 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:06.018826    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 12:43:06.072082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:06.071555    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 12:43:06.072082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:06.071611    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 12:43:06.344864 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:06.343905    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:43:06.500155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:06.500126    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:43:07.019069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:07.019022    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 12:43:07.019564 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:07.019099    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 12:43:07.072077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:07.070876    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 12:43:07.072077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:07.070934    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 12:43:07.183080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:07.181643    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:43:08.022076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:08.018221    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 12:43:08.022076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:08.018284    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 12:43:08.070969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:08.070933    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 12:43:08.071252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:08.071224    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 12:43:09.018497 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:09.018462    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 01 12:43:09.018709 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:09.018520    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 01 12:43:09.068990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:09.068949    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 01 12:43:09.069547 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:09.069015    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 01 12:43:10.024201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:10.024162    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:10.024701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:10.024674    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:10.025410 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:10.025392    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:43:10.074898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:10.074861    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:10.075481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:10.075289    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:10.076264 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:10.076243    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:43:13.753820 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.753769    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"]
Nov 01 12:43:13.760755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.760718    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:13.808794 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.808755    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"]
Nov 01 12:43:13.809772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.809746    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7l9\" (UniqueName: \"kubernetes.io/projected/8f096267-c0f2-4ef1-b356-5fc51bdc8602-kube-api-access-8c7l9\") pod \"maas-api-revoke-sub-bdb64dbf-xll6l\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:13.809869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.809805    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb64dbf-xll6l\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:13.809930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.809870    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb64dbf-xll6l\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:13.847476 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.847399    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:43:13.847677 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.847649    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:43:13.912239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.912199    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7l9\" (UniqueName: \"kubernetes.io/projected/8f096267-c0f2-4ef1-b356-5fc51bdc8602-kube-api-access-8c7l9\") pod \"maas-api-revoke-sub-bdb64dbf-xll6l\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:13.912239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.912244    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb64dbf-xll6l\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:13.912522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.912261    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb64dbf-xll6l\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:13.913266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.913241    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb64dbf-xll6l\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:13.914890 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.914858    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb64dbf-xll6l\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:13.967518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:13.967476    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7l9\" (UniqueName: \"kubernetes.io/projected/8f096267-c0f2-4ef1-b356-5fc51bdc8602-kube-api-access-8c7l9\") pod \"maas-api-revoke-sub-bdb64dbf-xll6l\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:14.089111 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:14.089012    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:14.490416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:14.490380    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"]
Nov 01 12:43:14.494446 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:43:14.494412    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f096267_c0f2_4ef1_b356_5fc51bdc8602.slice/crio-67007ef3f230eba1382d9a91c74f0f3e30d228b499c12309f25b9b1fdbf10a13 WatchSource:0}: Error finding container 67007ef3f230eba1382d9a91c74f0f3e30d228b499c12309f25b9b1fdbf10a13: Status 404 returned error can't find the container with id 67007ef3f230eba1382d9a91c74f0f3e30d228b499c12309f25b9b1fdbf10a13
Nov 01 12:43:15.098563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:15.098468    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"]
Nov 01 12:43:15.099152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:15.099131    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a04b2e-sv7np"]
Nov 01 12:43:15.229101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:15.229037    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l" event={"ID":"8f096267-c0f2-4ef1-b356-5fc51bdc8602","Type":"ContainerStarted","Data":"643cc7b3c27d9f9f01f0b654793b625061245b34c5ea66fdba6c22c186bb322b"}
Nov 01 12:43:15.229374 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:15.229357    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l" event={"ID":"8f096267-c0f2-4ef1-b356-5fc51bdc8602","Type":"ContainerStarted","Data":"67007ef3f230eba1382d9a91c74f0f3e30d228b499c12309f25b9b1fdbf10a13"}
Nov 01 12:43:15.363785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:15.363654    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l" podStartSLOduration=2.363631444 podStartE2EDuration="2.363631444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:15.353744971 +0000 UTC m=+1236.860936189" watchObservedRunningTime="2026-11-01 12:43:15.363631444 +0000 UTC m=+1236.870822662"
Nov 01 12:43:15.986140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:15.984544    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Nov 01 12:43:16.324387 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:16.324299    5725 generic.go:356] "Generic (PLEG): container finished" podID="8f096267-c0f2-4ef1-b356-5fc51bdc8602" containerID="643cc7b3c27d9f9f01f0b654793b625061245b34c5ea66fdba6c22c186bb322b" exitCode=0
Nov 01 12:43:16.324926 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:16.324906    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l" event={"ID":"8f096267-c0f2-4ef1-b356-5fc51bdc8602","Type":"ContainerDied","Data":"643cc7b3c27d9f9f01f0b654793b625061245b34c5ea66fdba6c22c186bb322b"}
Nov 01 12:43:16.680840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:16.680757    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d063d9-24ce-48bc-b915-4e21b494baba" path="/var/lib/kubelet/pods/48d063d9-24ce-48bc-b915-4e21b494baba/volumes"
Nov 01 12:43:17.539186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:17.539139    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.947761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:17.946549    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:43:17.970991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:17.970144    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:18.084145 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:18.073478    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:18.127164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:18.124154    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:18.267616 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:18.267569    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:43:18.648124 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:18.647560    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:43:18.811075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:18.810943    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:43:18.983156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:18.955066    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:19.013075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:18.983574    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:43:19.056036 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.055995    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:19.200230 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.200176    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:19.228474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.107518    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:19.228474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.203102    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:43:19.228474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.203191    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:43:19.228474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.203247    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:43:19.296153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.274650    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:43:19.341909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.341876    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:43:19.485608 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.485573    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:19.700134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.699915    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-openshift-ca\") pod \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") "
Nov 01 12:43:19.700134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.699990    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-xks-ca\") pod \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") "
Nov 01 12:43:19.700662 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.700319    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f096267-c0f2-4ef1-b356-5fc51bdc8602-kube-api-access-8c7l9\" (UniqueName: \"kubernetes.io/projected/8f096267-c0f2-4ef1-b356-5fc51bdc8602-kube-api-access-8c7l9\") pod \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\" (UID: \"8f096267-c0f2-4ef1-b356-5fc51bdc8602\") "
Nov 01 12:43:19.713380 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.713347    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:43:19.727082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.719924    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-openshift-ca" pod "8f096267-c0f2-4ef1-b356-5fc51bdc8602" (UID: "8f096267-c0f2-4ef1-b356-5fc51bdc8602"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:43:19.727337 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.727303    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:43:19.823075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.812637    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:19.940643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:19.940604    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-xks-ca" pod "8f096267-c0f2-4ef1-b356-5fc51bdc8602" (UID: "8f096267-c0f2-4ef1-b356-5fc51bdc8602"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:43:20.049076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.013861    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f096267-c0f2-4ef1-b356-5fc51bdc8602-kube-api-access-8c7l9" pod "8f096267-c0f2-4ef1-b356-5fc51bdc8602" (UID: "8f096267-c0f2-4ef1-b356-5fc51bdc8602"). InnerVolumeSpecName "kube-api-access-8c7l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:43:20.066076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.050309    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7l9\" (UniqueName: \"kubernetes.io/projected/8f096267-c0f2-4ef1-b356-5fc51bdc8602-kube-api-access-8c7l9\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:20.095077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.070103    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f096267-c0f2-4ef1-b356-5fc51bdc8602-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:20.168227 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.168169    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:20.245526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.240854    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:20.245526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.241224    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:20.367362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.367243    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:20.394207 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.394169    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"
Nov 01 12:43:20.410149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.410110    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l" event={"ID":"8f096267-c0f2-4ef1-b356-5fc51bdc8602","Type":"ContainerDied","Data":"67007ef3f230eba1382d9a91c74f0f3e30d228b499c12309f25b9b1fdbf10a13"}
Nov 01 12:43:20.422930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.410786    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67007ef3f230eba1382d9a91c74f0f3e30d228b499c12309f25b9b1fdbf10a13"
Nov 01 12:43:20.433978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.429416    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:43:20.453029 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.453002    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:43:20.738540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:20.738505    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:43:21.114873 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:21.114313    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:43:21.116375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:21.116345    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:43:23.045753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:23.045708    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"]
Nov 01 12:43:23.092325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:23.092292    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0310f236-fsvll"]
Nov 01 12:43:24.617373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:24.617336    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea5ad83-0931-4ed3-86d2-212132864725" path="/var/lib/kubelet/pods/6ea5ad83-0931-4ed3-86d2-212132864725/volumes"
Nov 01 12:43:29.869214 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:29.869177    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"]
Nov 01 12:43:29.869829 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:29.869809    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8f096267-c0f2-4ef1-b356-5fc51bdc8602" containerName="revoke-keys"
Nov 01 12:43:29.869829 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:29.869827    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f096267-c0f2-4ef1-b356-5fc51bdc8602" containerName="revoke-keys"
Nov 01 12:43:29.870006 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:29.869976    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f096267-c0f2-4ef1-b356-5fc51bdc8602" containerName="revoke-keys"
Nov 01 12:43:29.873717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:29.873695    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:29.922778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:29.922747    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:43:29.922984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:29.922805    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:43:29.926327 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:29.926300    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"]
Nov 01 12:43:30.031672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.031639    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e839a2-97e8-490f-9de6-98349829b929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ed61bd-jbllh\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.031884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.031725    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8ggn\" (UniqueName: \"kubernetes.io/projected/28e839a2-97e8-490f-9de6-98349829b929-kube-api-access-g8ggn\") pod \"maas-api-revoke-sub-e0ed61bd-jbllh\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.031884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.031817    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e839a2-97e8-490f-9de6-98349829b929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ed61bd-jbllh\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.132988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.132904    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e839a2-97e8-490f-9de6-98349829b929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ed61bd-jbllh\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.133201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.132997    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ggn\" (UniqueName: \"kubernetes.io/projected/28e839a2-97e8-490f-9de6-98349829b929-kube-api-access-g8ggn\") pod \"maas-api-revoke-sub-e0ed61bd-jbllh\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.133201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.133035    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e839a2-97e8-490f-9de6-98349829b929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ed61bd-jbllh\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.133778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.133757    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e839a2-97e8-490f-9de6-98349829b929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ed61bd-jbllh\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.135430 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.135401    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e839a2-97e8-490f-9de6-98349829b929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ed61bd-jbllh\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.188239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.188209    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ggn\" (UniqueName: \"kubernetes.io/projected/28e839a2-97e8-490f-9de6-98349829b929-kube-api-access-g8ggn\") pod \"maas-api-revoke-sub-e0ed61bd-jbllh\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.488108 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.488038    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:30.879926 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.879842    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"]
Nov 01 12:43:30.887622 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:43:30.887590    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e839a2_97e8_490f_9de6_98349829b929.slice/crio-0d7c08355f837511b217118fe03b1c964eb560761dce195fbc9dcccd21cfe283 WatchSource:0}: Error finding container 0d7c08355f837511b217118fe03b1c964eb560761dce195fbc9dcccd21cfe283: Status 404 returned error can't find the container with id 0d7c08355f837511b217118fe03b1c964eb560761dce195fbc9dcccd21cfe283
Nov 01 12:43:30.987392 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.987359    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"]
Nov 01 12:43:30.993406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:30.993378    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.031548 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:43:31.031512    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:43:31.045181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.045153    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.045295 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.045200    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.045295 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.045284    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.045411 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.045317    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.045411 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.045385    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.045492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.045459    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.045492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.045475    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.045569 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.045526    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d18b955-e42c-4c05-92ba-f1ab13f98811-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.045569 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.045549    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wb47\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-kube-api-access-9wb47\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.048502 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.048474    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"]
Nov 01 12:43:31.095261 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.095222    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eda7ad1-pr9z2"]
Nov 01 12:43:31.099920 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.099884    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"]
Nov 01 12:43:31.146552 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146466    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.146552 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146506    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.146771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146574    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d18b955-e42c-4c05-92ba-f1ab13f98811-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.146771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146608    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wb47\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-kube-api-access-9wb47\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.146771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146654    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.146771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146694    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.146967 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146789    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.146967 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146833    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.146967 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146924    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.146967 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146922    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.147214 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.146986    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.147310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.147287    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.147384 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.147367    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d18b955-e42c-4c05-92ba-f1ab13f98811-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.147447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.147376    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.149403 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.149374    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.149742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.149719    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.200864 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.200829    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.201019 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.200847    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wb47\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-kube-api-access-9wb47\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.308666 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.308624    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:31.547621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.547586    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh" event={"ID":"28e839a2-97e8-490f-9de6-98349829b929","Type":"ContainerStarted","Data":"1b9ad4a8088bb7a7d4fc983945beabebfe66b1fd348ab5c8a71645355da0f38c"}
Nov 01 12:43:31.547833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.547666    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh" event={"ID":"28e839a2-97e8-490f-9de6-98349829b929","Type":"ContainerStarted","Data":"0d7c08355f837511b217118fe03b1c964eb560761dce195fbc9dcccd21cfe283"}
Nov 01 12:43:31.655123 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.655039    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh" podStartSLOduration=2.655026286 podStartE2EDuration="2.655026286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:31.653822921 +0000 UTC m=+1253.161014138" watchObservedRunningTime="2026-11-01 12:43:31.655026286 +0000 UTC m=+1253.162217502"
Nov 01 12:43:31.753087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.753034    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"]
Nov 01 12:43:31.758635 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:43:31.758595    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d18b955_e42c_4c05_92ba_f1ab13f98811.slice/crio-94ad9d75fcab42fb056ba24965b9a91ec7b8309c74c0d76e338aa1bdf2fd0163 WatchSource:0}: Error finding container 94ad9d75fcab42fb056ba24965b9a91ec7b8309c74c0d76e338aa1bdf2fd0163: Status 404 returned error can't find the container with id 94ad9d75fcab42fb056ba24965b9a91ec7b8309c74c0d76e338aa1bdf2fd0163
Nov 01 12:43:31.761378 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.761337    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:31.761455 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.761432    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:31.761489 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:31.761473    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:32.570962 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:32.570925    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" event={"ID":"6d18b955-e42c-4c05-92ba-f1ab13f98811","Type":"ContainerStarted","Data":"93b56ee831d68a33c28d36aa93460143088e1e88a3a2a879ed5924de9a4c49e4"}
Nov 01 12:43:32.571460 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:32.570983    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" event={"ID":"6d18b955-e42c-4c05-92ba-f1ab13f98811","Type":"ContainerStarted","Data":"94ad9d75fcab42fb056ba24965b9a91ec7b8309c74c0d76e338aa1bdf2fd0163"}
Nov 01 12:43:32.572880 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:32.572853    5725 generic.go:356] "Generic (PLEG): container finished" podID="28e839a2-97e8-490f-9de6-98349829b929" containerID="1b9ad4a8088bb7a7d4fc983945beabebfe66b1fd348ab5c8a71645355da0f38c" exitCode=0
Nov 01 12:43:32.573010 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:32.572935    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh" event={"ID":"28e839a2-97e8-490f-9de6-98349829b929","Type":"ContainerDied","Data":"1b9ad4a8088bb7a7d4fc983945beabebfe66b1fd348ab5c8a71645355da0f38c"}
Nov 01 12:43:32.618405 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:32.618368    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34016853-e091-4d88-a644-a8b2aab4935d" path="/var/lib/kubelet/pods/34016853-e091-4d88-a644-a8b2aab4935d/volumes"
Nov 01 12:43:32.778821 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:32.778767    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podStartSLOduration=2.778752934 podStartE2EDuration="2.778752934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:32.727372384 +0000 UTC m=+1254.234563602" watchObservedRunningTime="2026-11-01 12:43:32.778752934 +0000 UTC m=+1254.285944221"
Nov 01 12:43:33.210537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.210503    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"]
Nov 01 12:43:33.215538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.215517    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.277198 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.277162    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.277408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.277216    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.277408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.277312    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.277408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.277401    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.277565 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.277490    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.277613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.277576    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.277655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.277626    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5fp\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-kube-api-access-8c5fp\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.277705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.277652    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.277705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.277687    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.309258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.309223    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:33.310659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.310626    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 12:43:33.310821 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.310681    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 12:43:33.345075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.324513    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"]
Nov 01 12:43:33.379128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379092    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379180    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379218    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379280    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379357    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379428    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379462    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5fp\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-kube-api-access-8c5fp\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379499    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379551    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379620    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.379978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.379955    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.380679 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.380652    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.380781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.380698    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.380922 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.380899    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.382938 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.382914    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.387386 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.387361    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.458459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.458392    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.459424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.459368    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5fp\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-kube-api-access-8c5fp\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.558688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.558463    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:33.896684 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.896655    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:33.995231 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.995193    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e839a2-97e8-490f-9de6-98349829b929-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e839a2-97e8-490f-9de6-98349829b929-maas-api-xks-ca\") pod \"28e839a2-97e8-490f-9de6-98349829b929\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") "
Nov 01 12:43:33.995424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.995262    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e839a2-97e8-490f-9de6-98349829b929-kube-api-access-g8ggn\" (UniqueName: \"kubernetes.io/projected/28e839a2-97e8-490f-9de6-98349829b929-kube-api-access-g8ggn\") pod \"28e839a2-97e8-490f-9de6-98349829b929\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") "
Nov 01 12:43:33.995424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.995400    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e839a2-97e8-490f-9de6-98349829b929-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e839a2-97e8-490f-9de6-98349829b929-maas-api-openshift-ca\") pod \"28e839a2-97e8-490f-9de6-98349829b929\" (UID: \"28e839a2-97e8-490f-9de6-98349829b929\") "
Nov 01 12:43:33.995875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.995845    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e839a2-97e8-490f-9de6-98349829b929-maas-api-openshift-ca" pod "28e839a2-97e8-490f-9de6-98349829b929" (UID: "28e839a2-97e8-490f-9de6-98349829b929"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:43:33.997993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.997964    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e839a2-97e8-490f-9de6-98349829b929-kube-api-access-g8ggn" pod "28e839a2-97e8-490f-9de6-98349829b929" (UID: "28e839a2-97e8-490f-9de6-98349829b929"). InnerVolumeSpecName "kube-api-access-g8ggn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:43:33.998138 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:33.998013    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e839a2-97e8-490f-9de6-98349829b929-maas-api-xks-ca" pod "28e839a2-97e8-490f-9de6-98349829b929" (UID: "28e839a2-97e8-490f-9de6-98349829b929"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:43:34.083640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.083606    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"]
Nov 01 12:43:34.091568 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.091533    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"]
Nov 01 12:43:34.100178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.099877    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b26e86f-pdz9k"]
Nov 01 12:43:34.101556 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:43:34.100536    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9dea2be_d11e_4622_8bcf_b9da24ce773d.slice/crio-15719f821c260bf18941e1063121db14302a75bd1b3d05c021d419009c466fc8 WatchSource:0}: Error finding container 15719f821c260bf18941e1063121db14302a75bd1b3d05c021d419009c466fc8: Status 404 returned error can't find the container with id 15719f821c260bf18941e1063121db14302a75bd1b3d05c021d419009c466fc8
Nov 01 12:43:34.101556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.100837    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e839a2-97e8-490f-9de6-98349829b929-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:34.101735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.101564    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e839a2-97e8-490f-9de6-98349829b929-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:34.101735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.101583    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8ggn\" (UniqueName: \"kubernetes.io/projected/28e839a2-97e8-490f-9de6-98349829b929-kube-api-access-g8ggn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:34.103790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.103736    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:34.103899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.103833    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:34.103899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.103879    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:34.309601 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.309567    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 12:43:34.309777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.309617    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 12:43:34.595819 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.595759    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" event={"ID":"a9dea2be-d11e-4622-8bcf-b9da24ce773d","Type":"ContainerStarted","Data":"2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc"}
Nov 01 12:43:34.596015 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.595835    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" event={"ID":"a9dea2be-d11e-4622-8bcf-b9da24ce773d","Type":"ContainerStarted","Data":"15719f821c260bf18941e1063121db14302a75bd1b3d05c021d419009c466fc8"}
Nov 01 12:43:34.597360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.597335    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh" event={"ID":"28e839a2-97e8-490f-9de6-98349829b929","Type":"ContainerDied","Data":"0d7c08355f837511b217118fe03b1c964eb560761dce195fbc9dcccd21cfe283"}
Nov 01 12:43:34.597479 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.597368    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"
Nov 01 12:43:34.597479 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.597389    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d7c08355f837511b217118fe03b1c964eb560761dce195fbc9dcccd21cfe283"
Nov 01 12:43:34.619164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.619134    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8294a4c4-14f9-4713-924d-7c7a1528f597" path="/var/lib/kubelet/pods/8294a4c4-14f9-4713-924d-7c7a1528f597/volumes"
Nov 01 12:43:34.713452 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:34.713404    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" podStartSLOduration=1.713390744 podStartE2EDuration="1.713390744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:34.711310057 +0000 UTC m=+1256.218501275" watchObservedRunningTime="2026-11-01 12:43:34.713390744 +0000 UTC m=+1256.220581960"
Nov 01 12:43:35.309127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:35.309089    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 12:43:35.309571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:35.309148    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 12:43:35.566133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:35.565781    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:35.566133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:35.565961    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 12:43:35.566133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:35.566006    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 12:43:36.309015 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:36.308979    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 12:43:36.309234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:36.309032    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 12:43:36.564129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:36.563423    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 12:43:36.564129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:36.563482    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 12:43:37.309845 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:37.309810    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 12:43:37.310283 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:37.309873    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 12:43:37.560130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:37.559604    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 12:43:37.560130 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:37.559680    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 12:43:38.309257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:38.309219    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 12:43:38.310002 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:38.309300    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 12:43:38.559587 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:38.559499    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 12:43:38.560247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:38.559836    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 12:43:39.575075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:39.564890    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 12:43:39.575075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:39.564950    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 12:43:39.613834 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:39.613802    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:39.616488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:39.616463    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:39.617699 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:39.617684    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:43:40.558902 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:40.558852    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 01 12:43:40.559134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:40.558928    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 01 12:43:41.831228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:41.831196    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:41.832235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:41.832197    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:41.833125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:41.833105    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:43:42.949760 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:42.949713    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"]
Nov 01 12:43:42.950739 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:42.950715    5725 cpu_manager.go:414] "containerMap: removing container" podUID="28e839a2-97e8-490f-9de6-98349829b929" containerName="revoke-keys"
Nov 01 12:43:42.950839 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:42.950741    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e839a2-97e8-490f-9de6-98349829b929" containerName="revoke-keys"
Nov 01 12:43:42.950952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:42.950935    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e839a2-97e8-490f-9de6-98349829b929" containerName="revoke-keys"
Nov 01 12:43:42.957012 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:42.956990    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.022241 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.022199    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"]
Nov 01 12:43:43.085633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.085599    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:43:43.085857 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.085728    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:43:43.094551 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.094518    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d4d6e8d-l2smg\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.094705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.094644    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ltj\" (UniqueName: \"kubernetes.io/projected/33a213c0-bc4e-4e52-b254-0e0776a0290f-kube-api-access-m4ltj\") pod \"maas-api-revoke-sub-7d4d6e8d-l2smg\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.094747 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.094707    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d4d6e8d-l2smg\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.196557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.196519    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d4d6e8d-l2smg\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.196790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.196633    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ltj\" (UniqueName: \"kubernetes.io/projected/33a213c0-bc4e-4e52-b254-0e0776a0290f-kube-api-access-m4ltj\") pod \"maas-api-revoke-sub-7d4d6e8d-l2smg\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.196790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.196669    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d4d6e8d-l2smg\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.197431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.197410    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d4d6e8d-l2smg\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.198984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.198960    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d4d6e8d-l2smg\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.254851 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.254785    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ltj\" (UniqueName: \"kubernetes.io/projected/33a213c0-bc4e-4e52-b254-0e0776a0290f-kube-api-access-m4ltj\") pod \"maas-api-revoke-sub-7d4d6e8d-l2smg\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.272865 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.272833    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:43.678119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.677087    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"]
Nov 01 12:43:43.683042 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:43:43.683016    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a213c0_bc4e_4e52_b254_0e0776a0290f.slice/crio-42afae09bb92cb1ed853eb5be2dc5a3f1310cf894396cf8d0a14747d9762858a WatchSource:0}: Error finding container 42afae09bb92cb1ed853eb5be2dc5a3f1310cf894396cf8d0a14747d9762858a: Status 404 returned error can't find the container with id 42afae09bb92cb1ed853eb5be2dc5a3f1310cf894396cf8d0a14747d9762858a
Nov 01 12:43:43.769388 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:43.769352    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg" event={"ID":"33a213c0-bc4e-4e52-b254-0e0776a0290f","Type":"ContainerStarted","Data":"42afae09bb92cb1ed853eb5be2dc5a3f1310cf894396cf8d0a14747d9762858a"}
Nov 01 12:43:44.778987 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:44.778949    5725 generic.go:356] "Generic (PLEG): container finished" podID="33a213c0-bc4e-4e52-b254-0e0776a0290f" containerID="810e5b051ffd156805f3999421d5ffb95a33c1e23f5a17993fb1f7b71d2ea777" exitCode=0
Nov 01 12:43:44.779497 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:44.779027    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg" event={"ID":"33a213c0-bc4e-4e52-b254-0e0776a0290f","Type":"ContainerDied","Data":"810e5b051ffd156805f3999421d5ffb95a33c1e23f5a17993fb1f7b71d2ea777"}
Nov 01 12:43:45.998567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:45.998539    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:46.031330 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.031291    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-openshift-ca\") pod \"33a213c0-bc4e-4e52-b254-0e0776a0290f\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") "
Nov 01 12:43:46.031533 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.031425    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a213c0-bc4e-4e52-b254-0e0776a0290f-kube-api-access-m4ltj\" (UniqueName: \"kubernetes.io/projected/33a213c0-bc4e-4e52-b254-0e0776a0290f-kube-api-access-m4ltj\") pod \"33a213c0-bc4e-4e52-b254-0e0776a0290f\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") "
Nov 01 12:43:46.031625 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.031605    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-xks-ca\") pod \"33a213c0-bc4e-4e52-b254-0e0776a0290f\" (UID: \"33a213c0-bc4e-4e52-b254-0e0776a0290f\") "
Nov 01 12:43:46.031771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.031732    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-openshift-ca" pod "33a213c0-bc4e-4e52-b254-0e0776a0290f" (UID: "33a213c0-bc4e-4e52-b254-0e0776a0290f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:43:46.032173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.032145    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:46.033715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.033686    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-xks-ca" pod "33a213c0-bc4e-4e52-b254-0e0776a0290f" (UID: "33a213c0-bc4e-4e52-b254-0e0776a0290f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:43:46.034153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.034135    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a213c0-bc4e-4e52-b254-0e0776a0290f-kube-api-access-m4ltj" pod "33a213c0-bc4e-4e52-b254-0e0776a0290f" (UID: "33a213c0-bc4e-4e52-b254-0e0776a0290f"). InnerVolumeSpecName "kube-api-access-m4ltj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:43:46.134233 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.134192    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4ltj\" (UniqueName: \"kubernetes.io/projected/33a213c0-bc4e-4e52-b254-0e0776a0290f-kube-api-access-m4ltj\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:46.134233 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.134233    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a213c0-bc4e-4e52-b254-0e0776a0290f-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:46.798679 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.798647    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"
Nov 01 12:43:46.798865 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.798651    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg" event={"ID":"33a213c0-bc4e-4e52-b254-0e0776a0290f","Type":"ContainerDied","Data":"42afae09bb92cb1ed853eb5be2dc5a3f1310cf894396cf8d0a14747d9762858a"}
Nov 01 12:43:46.798865 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:46.798774    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42afae09bb92cb1ed853eb5be2dc5a3f1310cf894396cf8d0a14747d9762858a"
Nov 01 12:43:47.156885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.156845    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"]
Nov 01 12:43:47.157748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.157722    5725 cpu_manager.go:414] "containerMap: removing container" podUID="33a213c0-bc4e-4e52-b254-0e0776a0290f" containerName="revoke-keys"
Nov 01 12:43:47.157849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.157750    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a213c0-bc4e-4e52-b254-0e0776a0290f" containerName="revoke-keys"
Nov 01 12:43:47.157980 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.157966    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a213c0-bc4e-4e52-b254-0e0776a0290f" containerName="revoke-keys"
Nov 01 12:43:47.161933 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.161908    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.212418 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.212386    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:43:47.212840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.212818    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:43:47.215223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.215194    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"]
Nov 01 12:43:47.253492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.253451    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30136c3e-gvk22\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.253781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.253756    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clhkm\" (UniqueName: \"kubernetes.io/projected/08ea3116-f940-48fc-9f4b-811dab290f3d-kube-api-access-clhkm\") pod \"maas-api-revoke-sub-30136c3e-gvk22\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.254070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.254034    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30136c3e-gvk22\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.355729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.355686    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30136c3e-gvk22\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.355930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.355908    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30136c3e-gvk22\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.355987 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.355971    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clhkm\" (UniqueName: \"kubernetes.io/projected/08ea3116-f940-48fc-9f4b-811dab290f3d-kube-api-access-clhkm\") pod \"maas-api-revoke-sub-30136c3e-gvk22\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.356781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.356750    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30136c3e-gvk22\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.358211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.358189    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30136c3e-gvk22\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.409444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.409372    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clhkm\" (UniqueName: \"kubernetes.io/projected/08ea3116-f940-48fc-9f4b-811dab290f3d-kube-api-access-clhkm\") pod \"maas-api-revoke-sub-30136c3e-gvk22\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.479521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.479489    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:47.877570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:47.877529    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"]
Nov 01 12:43:47.882724 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:43:47.882692    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ea3116_f940_48fc_9f4b_811dab290f3d.slice/crio-d238093e680e1cf2a29bff371324aec6617a7606edb14040e85c4cdf92598db2 WatchSource:0}: Error finding container d238093e680e1cf2a29bff371324aec6617a7606edb14040e85c4cdf92598db2: Status 404 returned error can't find the container with id d238093e680e1cf2a29bff371324aec6617a7606edb14040e85c4cdf92598db2
Nov 01 12:43:48.377075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.372630    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"]
Nov 01 12:43:48.425100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.421315    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.511442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.496285    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.511442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.496398    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dae956-54ad-41d5-8441-f9fec077acfd-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.511442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.496455    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.511442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.496489    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.511442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.496518    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.511442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.496606    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.511442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.496687    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.511442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.496732    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.511442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.496770    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrz4h\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-kube-api-access-wrz4h\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.605089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.605020    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.609211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.609181    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.609459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.609435    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrz4h\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-kube-api-access-wrz4h\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.609724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.609686    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.609915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.609899    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dae956-54ad-41d5-8441-f9fec077acfd-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.610067 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.610035    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.610214 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.610199    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.610331 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.610318    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.610502 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.610488    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.611101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.611042    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.611237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.605339    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"]
Nov 01 12:43:48.622402 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.622365    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.623747 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.622822    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.628238 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.605875    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.632219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.632188    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dae956-54ad-41d5-8441-f9fec077acfd-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.759567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.743510    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.759567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.744478    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrz4h\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-kube-api-access-wrz4h\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.773793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.768914    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.777088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.776997    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:48.831458 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.831353    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22" event={"ID":"08ea3116-f940-48fc-9f4b-811dab290f3d","Type":"ContainerStarted","Data":"fdbed3f8f1140ab3cef12a88633821c657f0332f87731701e1fbd28bb41fd698"}
Nov 01 12:43:48.832572 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:48.832542    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22" event={"ID":"08ea3116-f940-48fc-9f4b-811dab290f3d","Type":"ContainerStarted","Data":"d238093e680e1cf2a29bff371324aec6617a7606edb14040e85c4cdf92598db2"}
Nov 01 12:43:49.020310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:49.010576    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22" podStartSLOduration=2.010553622 podStartE2EDuration="2.010553622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:48.963331636 +0000 UTC m=+1270.470522855" watchObservedRunningTime="2026-11-01 12:43:49.010553622 +0000 UTC m=+1270.517744842"
Nov 01 12:43:49.020310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:49.019448    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:50.294863 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.292132    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:50.380596 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.377780    5725 generic.go:356] "Generic (PLEG): container finished" podID="08ea3116-f940-48fc-9f4b-811dab290f3d" containerID="fdbed3f8f1140ab3cef12a88633821c657f0332f87731701e1fbd28bb41fd698" exitCode=0
Nov 01 12:43:50.380596 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.378140    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22" event={"ID":"08ea3116-f940-48fc-9f4b-811dab290f3d","Type":"ContainerDied","Data":"fdbed3f8f1140ab3cef12a88633821c657f0332f87731701e1fbd28bb41fd698"}
Nov 01 12:43:50.381590 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.381562    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"]
Nov 01 12:43:50.431074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.430106    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:50.431074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.430197    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:50.431074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.430243    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:43:50.431074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.430927    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:43:50.894165 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.893710    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:43:50.895508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:50.894989    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:43:51.188073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.182199    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:51.311219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.281944    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:43:51.311876 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.292387    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:43:51.312594 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.312571    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:43:51.508795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.508755    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:51.531105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.511832    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:43:51.545455 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.545408    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" event={"ID":"f3dae956-54ad-41d5-8441-f9fec077acfd","Type":"ContainerStarted","Data":"43412775b7f57a4503f89d09962633d9c87d04c36c846545011387746965e9e5"}
Nov 01 12:43:51.627631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.627596    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:43:51.627925 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.627894    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:51.740912 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.740815    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:51.895374 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.895274    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:43:51.897029 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.897004    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:43:51.926076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.900217    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:51.928272 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:51.928244    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:43:52.194002 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.186125    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:52.197028 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.194788    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:43:52.292608 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.292566    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": context deadline exceeded"
Nov 01 12:43:52.313186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.313119    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:52.317437 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.317395    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:43:52.323510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.319918    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:43:52.432821 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.432793    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:43:52.515076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.510410    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:52.894748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.894026    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:52.896563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.896509    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:52.973076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.962719    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:43:53.003074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.998286    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-openshift-ca\") pod \"08ea3116-f940-48fc-9f4b-811dab290f3d\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") "
Nov 01 12:43:53.003074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.998526    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-xks-ca\") pod \"08ea3116-f940-48fc-9f4b-811dab290f3d\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") "
Nov 01 12:43:53.003074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:52.998570    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ea3116-f940-48fc-9f4b-811dab290f3d-kube-api-access-clhkm\" (UniqueName: \"kubernetes.io/projected/08ea3116-f940-48fc-9f4b-811dab290f3d-kube-api-access-clhkm\") pod \"08ea3116-f940-48fc-9f4b-811dab290f3d\" (UID: \"08ea3116-f940-48fc-9f4b-811dab290f3d\") "
Nov 01 12:43:53.004227 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.004194    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-openshift-ca" pod "08ea3116-f940-48fc-9f4b-811dab290f3d" (UID: "08ea3116-f940-48fc-9f4b-811dab290f3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:43:53.075268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.075217    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-xks-ca" pod "08ea3116-f940-48fc-9f4b-811dab290f3d" (UID: "08ea3116-f940-48fc-9f4b-811dab290f3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:43:53.158554 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.158462    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ea3116-f940-48fc-9f4b-811dab290f3d-kube-api-access-clhkm" pod "08ea3116-f940-48fc-9f4b-811dab290f3d" (UID: "08ea3116-f940-48fc-9f4b-811dab290f3d"). InnerVolumeSpecName "kube-api-access-clhkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:43:53.181014 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.180982    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:53.181272 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.181248    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ea3116-f940-48fc-9f4b-811dab290f3d-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:53.181375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.181363    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clhkm\" (UniqueName: \"kubernetes.io/projected/08ea3116-f940-48fc-9f4b-811dab290f3d-kube-api-access-clhkm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:53.202078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.200957    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Nov 01 12:43:53.223241 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.223199    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" event={"ID":"f3dae956-54ad-41d5-8441-f9fec077acfd","Type":"ContainerStarted","Data":"f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d"}
Nov 01 12:43:53.354075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.326316    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:43:53.382804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.382772    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:43:53.530264 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.530221    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:43:53.566451 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.566379    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podStartSLOduration=5.56635632 podStartE2EDuration="5.56635632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:53.383339573 +0000 UTC m=+1274.890530838" watchObservedRunningTime="2026-11-01 12:43:53.56635632 +0000 UTC m=+1275.073547539"
Nov 01 12:43:53.569309 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.569283    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:43:53.602415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.601148    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:43:53.602415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.601214    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:53.763186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.763154    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:43:53.786519 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.786435    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:43:53.832805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:53.832770    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:43:54.021642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:54.021602    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:43:54.022351 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:54.022324    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 12:43:54.022487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:54.022377    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 12:43:54.299197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:54.299168    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"
Nov 01 12:43:54.299567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:54.299193    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22" event={"ID":"08ea3116-f940-48fc-9f4b-811dab290f3d","Type":"ContainerDied","Data":"d238093e680e1cf2a29bff371324aec6617a7606edb14040e85c4cdf92598db2"}
Nov 01 12:43:54.299664 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:54.299603    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d238093e680e1cf2a29bff371324aec6617a7606edb14040e85c4cdf92598db2"
Nov 01 12:43:55.020356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.020320    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 12:43:55.020963 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.020926    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 12:43:55.279332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.279230    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"]
Nov 01 12:43:55.279900 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.279886    5725 cpu_manager.go:414] "containerMap: removing container" podUID="08ea3116-f940-48fc-9f4b-811dab290f3d" containerName="revoke-keys"
Nov 01 12:43:55.279948 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.279901    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ea3116-f940-48fc-9f4b-811dab290f3d" containerName="revoke-keys"
Nov 01 12:43:55.280023 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.280014    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ea3116-f940-48fc-9f4b-811dab290f3d" containerName="revoke-keys"
Nov 01 12:43:55.283956 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.283938    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.309788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.309751    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e147669-m6mx9\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.309961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.309856    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e147669-m6mx9\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.309961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.309918    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtc6t\" (UniqueName: \"kubernetes.io/projected/8558ba68-b862-4810-8e98-970ea9decba4-kube-api-access-wtc6t\") pod \"maas-api-revoke-sub-7e147669-m6mx9\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.350075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.349260    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:43:55.355071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.353419    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:43:55.373106 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.372796    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"]
Nov 01 12:43:55.378082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.377095    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"]
Nov 01 12:43:55.382656 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.382631    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.411793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.411739    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e147669-m6mx9\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.411986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.411799    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a9e061-dljrn\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.411986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.411890    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a9e061-dljrn\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.411986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.411928    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e147669-m6mx9\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.412194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.412018    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtc6t\" (UniqueName: \"kubernetes.io/projected/8558ba68-b862-4810-8e98-970ea9decba4-kube-api-access-wtc6t\") pod \"maas-api-revoke-sub-7e147669-m6mx9\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.412194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.412162    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f2b9\" (UniqueName: \"kubernetes.io/projected/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-kube-api-access-7f2b9\") pod \"maas-api-revoke-sub-80a9e061-dljrn\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.413113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.413085    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e147669-m6mx9\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.414969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.414915    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e147669-m6mx9\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.461171 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.461133    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"]
Nov 01 12:43:55.469164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.469132    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtc6t\" (UniqueName: \"kubernetes.io/projected/8558ba68-b862-4810-8e98-970ea9decba4-kube-api-access-wtc6t\") pod \"maas-api-revoke-sub-7e147669-m6mx9\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.513683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.513642    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a9e061-dljrn\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.513891 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.513855    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f2b9\" (UniqueName: \"kubernetes.io/projected/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-kube-api-access-7f2b9\") pod \"maas-api-revoke-sub-80a9e061-dljrn\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.513993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.513978    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a9e061-dljrn\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.514454 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.514432    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a9e061-dljrn\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.516738 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.516709    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a9e061-dljrn\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.575761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.575687    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f2b9\" (UniqueName: \"kubernetes.io/projected/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-kube-api-access-7f2b9\") pod \"maas-api-revoke-sub-80a9e061-dljrn\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:55.598426 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.598398    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:55.701502 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:55.701472    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:56.020994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:56.020962    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 12:43:56.021515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:56.021013    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 12:43:56.024449 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:56.024419    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"]
Nov 01 12:43:56.076929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:56.076892    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"]
Nov 01 12:43:56.084999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:56.084971    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01bc3687-l69nx"]
Nov 01 12:43:56.163797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:56.163764    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"]
Nov 01 12:43:56.317437 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:56.317403    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9" event={"ID":"8558ba68-b862-4810-8e98-970ea9decba4","Type":"ContainerStarted","Data":"62dd2f08bff08134eb82b9f8c480240ec76ee566c307f1473ff5f1993b0629e7"}
Nov 01 12:43:56.319089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:56.319058    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn" event={"ID":"2cf2c770-a555-43bc-bbcf-3c6beb11d60b","Type":"ContainerStarted","Data":"71d134d4ce0c00b21bc7fccfc4e2872352748c35cb379086c3c49cc0578e6ca2"}
Nov 01 12:43:56.619807 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:56.619770    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc355f7-12e8-42a5-bf79-1b9ef6be7d96" path="/var/lib/kubelet/pods/8cc355f7-12e8-42a5-bf79-1b9ef6be7d96/volumes"
Nov 01 12:43:57.020165 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.020130    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 12:43:57.020361 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.020189    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 12:43:57.062555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.062514    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"]
Nov 01 12:43:57.072411 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.072379    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce01fd73-2pzbp"]
Nov 01 12:43:57.346299 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.345132    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn" event={"ID":"2cf2c770-a555-43bc-bbcf-3c6beb11d60b","Type":"ContainerStarted","Data":"eeba5c19e18484db9c472070723f094df2224a9368092fd94705adb080e148ca"}
Nov 01 12:43:57.357125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.356970    5725 generic.go:356] "Generic (PLEG): container finished" podID="8558ba68-b862-4810-8e98-970ea9decba4" containerID="90015c5a40ce9aaf33c8c9cf0ada4b364f85519d6c33aa409a4e44c7ec7aad36" exitCode=0
Nov 01 12:43:57.357569 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.357226    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9" event={"ID":"8558ba68-b862-4810-8e98-970ea9decba4","Type":"ContainerDied","Data":"90015c5a40ce9aaf33c8c9cf0ada4b364f85519d6c33aa409a4e44c7ec7aad36"}
Nov 01 12:43:57.569244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.569188    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn" podStartSLOduration=2.569172607 podStartE2EDuration="2.569172607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:57.458728947 +0000 UTC m=+1278.965920165" watchObservedRunningTime="2026-11-01 12:43:57.569172607 +0000 UTC m=+1279.076363830"
Nov 01 12:43:57.959092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.957877    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"]
Nov 01 12:43:57.966445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.966417    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:57.993504 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:57.993464    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"]
Nov 01 12:43:58.020375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.020342    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 12:43:58.020550 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.020392    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 12:43:58.077532 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.077494    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9882a575-bppwz\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.077532 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.077533    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9882a575-bppwz\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.078141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.077589    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76krn\" (UniqueName: \"kubernetes.io/projected/d96357a3-b6af-4d20-b0f1-c39472546a60-kube-api-access-76krn\") pod \"maas-api-revoke-sub-9882a575-bppwz\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.179185 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.179148    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9882a575-bppwz\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.179185 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.179195    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9882a575-bppwz\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.179507 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.179258    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76krn\" (UniqueName: \"kubernetes.io/projected/d96357a3-b6af-4d20-b0f1-c39472546a60-kube-api-access-76krn\") pod \"maas-api-revoke-sub-9882a575-bppwz\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.180301 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.180276    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9882a575-bppwz\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.181976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.181951    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9882a575-bppwz\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.250077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.234585    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76krn\" (UniqueName: \"kubernetes.io/projected/d96357a3-b6af-4d20-b0f1-c39472546a60-kube-api-access-76krn\") pod \"maas-api-revoke-sub-9882a575-bppwz\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.282106 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.282044    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:43:58.369969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.369930    5725 generic.go:356] "Generic (PLEG): container finished" podID="2cf2c770-a555-43bc-bbcf-3c6beb11d60b" containerID="eeba5c19e18484db9c472070723f094df2224a9368092fd94705adb080e148ca" exitCode=0
Nov 01 12:43:58.370197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.370095    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn" event={"ID":"2cf2c770-a555-43bc-bbcf-3c6beb11d60b","Type":"ContainerDied","Data":"eeba5c19e18484db9c472070723f094df2224a9368092fd94705adb080e148ca"}
Nov 01 12:43:58.678678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.676911    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d15fcb-133f-40dd-b873-929f1b71a460" path="/var/lib/kubelet/pods/b5d15fcb-133f-40dd-b873-929f1b71a460/volumes"
Nov 01 12:43:58.696904 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.696876    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:58.711234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.711196    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"]
Nov 01 12:43:58.716146 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:43:58.716115    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96357a3_b6af_4d20_b0f1_c39472546a60.slice/crio-47e271b1e6f6f9857c9f8cfb4e512d2a219c149d25d1640a3cf02946be57c9ed WatchSource:0}: Error finding container 47e271b1e6f6f9857c9f8cfb4e512d2a219c149d25d1640a3cf02946be57c9ed: Status 404 returned error can't find the container with id 47e271b1e6f6f9857c9f8cfb4e512d2a219c149d25d1640a3cf02946be57c9ed
Nov 01 12:43:58.791421 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.791391    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8558ba68-b862-4810-8e98-970ea9decba4-kube-api-access-wtc6t\" (UniqueName: \"kubernetes.io/projected/8558ba68-b862-4810-8e98-970ea9decba4-kube-api-access-wtc6t\") pod \"8558ba68-b862-4810-8e98-970ea9decba4\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") "
Nov 01 12:43:58.791567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.791471    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-xks-ca\") pod \"8558ba68-b862-4810-8e98-970ea9decba4\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") "
Nov 01 12:43:58.791672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.791655    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-openshift-ca\") pod \"8558ba68-b862-4810-8e98-970ea9decba4\" (UID: \"8558ba68-b862-4810-8e98-970ea9decba4\") "
Nov 01 12:43:58.792118 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.792092    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-openshift-ca" pod "8558ba68-b862-4810-8e98-970ea9decba4" (UID: "8558ba68-b862-4810-8e98-970ea9decba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:43:58.792613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.792592    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:58.793497 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.793476    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-xks-ca" pod "8558ba68-b862-4810-8e98-970ea9decba4" (UID: "8558ba68-b862-4810-8e98-970ea9decba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:43:58.793928 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.793907    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8558ba68-b862-4810-8e98-970ea9decba4-kube-api-access-wtc6t" pod "8558ba68-b862-4810-8e98-970ea9decba4" (UID: "8558ba68-b862-4810-8e98-970ea9decba4"). InnerVolumeSpecName "kube-api-access-wtc6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:43:58.893575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.893534    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtc6t\" (UniqueName: \"kubernetes.io/projected/8558ba68-b862-4810-8e98-970ea9decba4-kube-api-access-wtc6t\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:58.893575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:58.893568    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8558ba68-b862-4810-8e98-970ea9decba4-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:59.020200 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.020163    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 12:43:59.020391 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.020223    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 12:43:59.382299 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.382187    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz" event={"ID":"d96357a3-b6af-4d20-b0f1-c39472546a60","Type":"ContainerStarted","Data":"6dccd38bfa09c81883f4356026f298d766fa18b0db8ad37dbf589126e3d45f6b"}
Nov 01 12:43:59.382299 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.382264    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz" event={"ID":"d96357a3-b6af-4d20-b0f1-c39472546a60","Type":"ContainerStarted","Data":"47e271b1e6f6f9857c9f8cfb4e512d2a219c149d25d1640a3cf02946be57c9ed"}
Nov 01 12:43:59.386063 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.386027    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"
Nov 01 12:43:59.386226 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.386075    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9" event={"ID":"8558ba68-b862-4810-8e98-970ea9decba4","Type":"ContainerDied","Data":"62dd2f08bff08134eb82b9f8c480240ec76ee566c307f1473ff5f1993b0629e7"}
Nov 01 12:43:59.386226 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.386137    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62dd2f08bff08134eb82b9f8c480240ec76ee566c307f1473ff5f1993b0629e7"
Nov 01 12:43:59.504247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.504012    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz" podStartSLOduration=2.50399875 podStartE2EDuration="2.50399875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:59.502324264 +0000 UTC m=+1281.009515482" watchObservedRunningTime="2026-11-01 12:43:59.50399875 +0000 UTC m=+1281.011189967"
Nov 01 12:43:59.587195 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.587165    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:43:59.707231 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.706179    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-kube-api-access-7f2b9\" (UniqueName: \"kubernetes.io/projected/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-kube-api-access-7f2b9\") pod \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") "
Nov 01 12:43:59.707231 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.706313    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-openshift-ca\") pod \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") "
Nov 01 12:43:59.707231 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.706419    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-xks-ca\") pod \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\" (UID: \"2cf2c770-a555-43bc-bbcf-3c6beb11d60b\") "
Nov 01 12:43:59.708330 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.708304    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-openshift-ca" pod "2cf2c770-a555-43bc-bbcf-3c6beb11d60b" (UID: "2cf2c770-a555-43bc-bbcf-3c6beb11d60b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:43:59.714461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.714429    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-kube-api-access-7f2b9" pod "2cf2c770-a555-43bc-bbcf-3c6beb11d60b" (UID: "2cf2c770-a555-43bc-bbcf-3c6beb11d60b"). InnerVolumeSpecName "kube-api-access-7f2b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:43:59.715300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.715247    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-xks-ca" pod "2cf2c770-a555-43bc-bbcf-3c6beb11d60b" (UID: "2cf2c770-a555-43bc-bbcf-3c6beb11d60b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:43:59.808761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.808719    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f2b9\" (UniqueName: \"kubernetes.io/projected/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-kube-api-access-7f2b9\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:59.808933 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.808773    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:43:59.808933 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:43:59.808791    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf2c770-a555-43bc-bbcf-3c6beb11d60b-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:00.023522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.023485    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 12:44:00.023740 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.023553    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 12:44:00.100927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.100882    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"]
Nov 01 12:44:00.101863 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.101835    5725 cpu_manager.go:414] "containerMap: removing container" podUID="2cf2c770-a555-43bc-bbcf-3c6beb11d60b" containerName="revoke-keys"
Nov 01 12:44:00.101863 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.101861    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf2c770-a555-43bc-bbcf-3c6beb11d60b" containerName="revoke-keys"
Nov 01 12:44:00.102146 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.101896    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8558ba68-b862-4810-8e98-970ea9decba4" containerName="revoke-keys"
Nov 01 12:44:00.102146 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.101903    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8558ba68-b862-4810-8e98-970ea9decba4" containerName="revoke-keys"
Nov 01 12:44:00.102146 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.102077    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8558ba68-b862-4810-8e98-970ea9decba4" containerName="revoke-keys"
Nov 01 12:44:00.102146 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.102100    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf2c770-a555-43bc-bbcf-3c6beb11d60b" containerName="revoke-keys"
Nov 01 12:44:00.105912 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.105892    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.157743 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.157705    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"]
Nov 01 12:44:00.217524 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.217479    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec0111d7-f7mtt\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.217746 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.217725    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec0111d7-f7mtt\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.217874 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.217851    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v855\" (UniqueName: \"kubernetes.io/projected/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-kube-api-access-2v855\") pod \"maas-api-revoke-sub-ec0111d7-f7mtt\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.320527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.319699    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec0111d7-f7mtt\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.320527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.319965    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec0111d7-f7mtt\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.320527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.320106    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v855\" (UniqueName: \"kubernetes.io/projected/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-kube-api-access-2v855\") pod \"maas-api-revoke-sub-ec0111d7-f7mtt\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.320838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.320552    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec0111d7-f7mtt\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.323184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.323142    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec0111d7-f7mtt\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.374644 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.374611    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v855\" (UniqueName: \"kubernetes.io/projected/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-kube-api-access-2v855\") pod \"maas-api-revoke-sub-ec0111d7-f7mtt\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.398397 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.398347    5725 generic.go:356] "Generic (PLEG): container finished" podID="d96357a3-b6af-4d20-b0f1-c39472546a60" containerID="6dccd38bfa09c81883f4356026f298d766fa18b0db8ad37dbf589126e3d45f6b" exitCode=0
Nov 01 12:44:00.398772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.398426    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz" event={"ID":"d96357a3-b6af-4d20-b0f1-c39472546a60","Type":"ContainerDied","Data":"6dccd38bfa09c81883f4356026f298d766fa18b0db8ad37dbf589126e3d45f6b"}
Nov 01 12:44:00.400280 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.400259    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"
Nov 01 12:44:00.400375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.400276    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn" event={"ID":"2cf2c770-a555-43bc-bbcf-3c6beb11d60b","Type":"ContainerDied","Data":"71d134d4ce0c00b21bc7fccfc4e2872352748c35cb379086c3c49cc0578e6ca2"}
Nov 01 12:44:00.400375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.400319    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71d134d4ce0c00b21bc7fccfc4e2872352748c35cb379086c3c49cc0578e6ca2"
Nov 01 12:44:00.427294 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.427268    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:00.850250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:00.850215    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"]
Nov 01 12:44:00.854661 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:00.854631    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb947bb64_ac64_42ea_9aac_3b4ae73d1e9a.slice/crio-044b01b087bffd417a03250544490ed0de3294405b05b897dfbb26550d5ff0f4 WatchSource:0}: Error finding container 044b01b087bffd417a03250544490ed0de3294405b05b897dfbb26550d5ff0f4: Status 404 returned error can't find the container with id 044b01b087bffd417a03250544490ed0de3294405b05b897dfbb26550d5ff0f4
Nov 01 12:44:01.026631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.026597    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:44:01.027067 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.027003    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:44:01.027936 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.027914    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:44:01.029451 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:44:01.029422    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:44:01.410951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.410911    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt" event={"ID":"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a","Type":"ContainerStarted","Data":"45115db1c3c41c22d9480e0b6d99dc0fc454191d0d3a4792357d2b4106efa2df"}
Nov 01 12:44:01.411448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.410978    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt" event={"ID":"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a","Type":"ContainerStarted","Data":"044b01b087bffd417a03250544490ed0de3294405b05b897dfbb26550d5ff0f4"}
Nov 01 12:44:01.623135 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.623110    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:44:01.737072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.736996    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt" podStartSLOduration=1.736972084 podStartE2EDuration="1.736972084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:01.522455888 +0000 UTC m=+1283.029647106" watchObservedRunningTime="2026-11-01 12:44:01.736972084 +0000 UTC m=+1283.244163302"
Nov 01 12:44:01.754632 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.754596    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96357a3-b6af-4d20-b0f1-c39472546a60-kube-api-access-76krn\" (UniqueName: \"kubernetes.io/projected/d96357a3-b6af-4d20-b0f1-c39472546a60-kube-api-access-76krn\") pod \"d96357a3-b6af-4d20-b0f1-c39472546a60\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") "
Nov 01 12:44:01.754850 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.754709    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-openshift-ca\") pod \"d96357a3-b6af-4d20-b0f1-c39472546a60\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") "
Nov 01 12:44:01.754850 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.754775    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-xks-ca\") pod \"d96357a3-b6af-4d20-b0f1-c39472546a60\" (UID: \"d96357a3-b6af-4d20-b0f1-c39472546a60\") "
Nov 01 12:44:01.755850 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.755819    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-openshift-ca" pod "d96357a3-b6af-4d20-b0f1-c39472546a60" (UID: "d96357a3-b6af-4d20-b0f1-c39472546a60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:01.756491 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.756466    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:01.758112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.758077    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96357a3-b6af-4d20-b0f1-c39472546a60-kube-api-access-76krn" pod "d96357a3-b6af-4d20-b0f1-c39472546a60" (UID: "d96357a3-b6af-4d20-b0f1-c39472546a60"). InnerVolumeSpecName "kube-api-access-76krn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:01.771957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.763177    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-xks-ca" pod "d96357a3-b6af-4d20-b0f1-c39472546a60" (UID: "d96357a3-b6af-4d20-b0f1-c39472546a60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:01.858173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.858084    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76krn\" (UniqueName: \"kubernetes.io/projected/d96357a3-b6af-4d20-b0f1-c39472546a60-kube-api-access-76krn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:01.858173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:01.858117    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96357a3-b6af-4d20-b0f1-c39472546a60-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:02.423461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:02.423427    5725 generic.go:356] "Generic (PLEG): container finished" podID="b947bb64-ac64-42ea-9aac-3b4ae73d1e9a" containerID="45115db1c3c41c22d9480e0b6d99dc0fc454191d0d3a4792357d2b4106efa2df" exitCode=0
Nov 01 12:44:02.423920 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:02.423509    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt" event={"ID":"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a","Type":"ContainerDied","Data":"45115db1c3c41c22d9480e0b6d99dc0fc454191d0d3a4792357d2b4106efa2df"}
Nov 01 12:44:02.425282 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:02.425253    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"
Nov 01 12:44:02.425436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:02.425285    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz" event={"ID":"d96357a3-b6af-4d20-b0f1-c39472546a60","Type":"ContainerDied","Data":"47e271b1e6f6f9857c9f8cfb4e512d2a219c149d25d1640a3cf02946be57c9ed"}
Nov 01 12:44:02.425436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:02.425333    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47e271b1e6f6f9857c9f8cfb4e512d2a219c149d25d1640a3cf02946be57c9ed"
Nov 01 12:44:03.602025 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.601996    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:03.683688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.683647    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-xks-ca\") pod \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") "
Nov 01 12:44:03.683890 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.683722    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-kube-api-access-2v855\" (UniqueName: \"kubernetes.io/projected/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-kube-api-access-2v855\") pod \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") "
Nov 01 12:44:03.683957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.683915    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-openshift-ca\") pod \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\" (UID: \"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a\") "
Nov 01 12:44:03.684342 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.684316    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-openshift-ca" pod "b947bb64-ac64-42ea-9aac-3b4ae73d1e9a" (UID: "b947bb64-ac64-42ea-9aac-3b4ae73d1e9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:03.686024 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.685965    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-xks-ca" pod "b947bb64-ac64-42ea-9aac-3b4ae73d1e9a" (UID: "b947bb64-ac64-42ea-9aac-3b4ae73d1e9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:03.686131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.686075    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-kube-api-access-2v855" pod "b947bb64-ac64-42ea-9aac-3b4ae73d1e9a" (UID: "b947bb64-ac64-42ea-9aac-3b4ae73d1e9a"). InnerVolumeSpecName "kube-api-access-2v855". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:03.785841 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.785796    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:03.785841 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.785835    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:03.785841 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:03.785844    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v855\" (UniqueName: \"kubernetes.io/projected/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a-kube-api-access-2v855\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:04.443659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:04.443624    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"
Nov 01 12:44:04.443659 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:04.443647    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt" event={"ID":"b947bb64-ac64-42ea-9aac-3b4ae73d1e9a","Type":"ContainerDied","Data":"044b01b087bffd417a03250544490ed0de3294405b05b897dfbb26550d5ff0f4"}
Nov 01 12:44:04.443886 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:04.443706    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="044b01b087bffd417a03250544490ed0de3294405b05b897dfbb26550d5ff0f4"
Nov 01 12:44:05.484336 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.484300    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"]
Nov 01 12:44:05.484963 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.484948    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b947bb64-ac64-42ea-9aac-3b4ae73d1e9a" containerName="revoke-keys"
Nov 01 12:44:05.485007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.484963    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b947bb64-ac64-42ea-9aac-3b4ae73d1e9a" containerName="revoke-keys"
Nov 01 12:44:05.485007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.484997    5725 cpu_manager.go:414] "containerMap: removing container" podUID="d96357a3-b6af-4d20-b0f1-c39472546a60" containerName="revoke-keys"
Nov 01 12:44:05.485007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.485002    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96357a3-b6af-4d20-b0f1-c39472546a60" containerName="revoke-keys"
Nov 01 12:44:05.485158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.485108    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b947bb64-ac64-42ea-9aac-3b4ae73d1e9a" containerName="revoke-keys"
Nov 01 12:44:05.485158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.485119    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96357a3-b6af-4d20-b0f1-c39472546a60" containerName="revoke-keys"
Nov 01 12:44:05.489136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.489117    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.542652 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.542619    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"]
Nov 01 12:44:05.581558 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.581525    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:44:05.581558 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.581556    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:44:05.612067 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.612029    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91d70d47-jfv6g\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.612242 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.612116    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91d70d47-jfv6g\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.612302 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.612278    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4bc\" (UniqueName: \"kubernetes.io/projected/0c576d1c-8cab-4947-b614-c4159156994e-kube-api-access-fx4bc\") pod \"maas-api-revoke-sub-91d70d47-jfv6g\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.713805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.713769    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91d70d47-jfv6g\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.714325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.714290    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4bc\" (UniqueName: \"kubernetes.io/projected/0c576d1c-8cab-4947-b614-c4159156994e-kube-api-access-fx4bc\") pod \"maas-api-revoke-sub-91d70d47-jfv6g\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.714565 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.714542    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91d70d47-jfv6g\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.714616 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.714597    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91d70d47-jfv6g\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.716936 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.716915    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91d70d47-jfv6g\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.769526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.769435    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4bc\" (UniqueName: \"kubernetes.io/projected/0c576d1c-8cab-4947-b614-c4159156994e-kube-api-access-fx4bc\") pod \"maas-api-revoke-sub-91d70d47-jfv6g\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:05.803200 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:05.803165    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:06.188795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:06.188706    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"]
Nov 01 12:44:06.193352 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:06.193322    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c576d1c_8cab_4947_b614_c4159156994e.slice/crio-1158bf048427a98a14ab2e29865611051818bee8172454e7a655f1a228502544 WatchSource:0}: Error finding container 1158bf048427a98a14ab2e29865611051818bee8172454e7a655f1a228502544: Status 404 returned error can't find the container with id 1158bf048427a98a14ab2e29865611051818bee8172454e7a655f1a228502544
Nov 01 12:44:06.468491 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:06.468399    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g" event={"ID":"0c576d1c-8cab-4947-b614-c4159156994e","Type":"ContainerStarted","Data":"1158bf048427a98a14ab2e29865611051818bee8172454e7a655f1a228502544"}
Nov 01 12:44:07.479561 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:07.479526    5725 generic.go:356] "Generic (PLEG): container finished" podID="0c576d1c-8cab-4947-b614-c4159156994e" containerID="201730e3268c927734d7e5a12eb9aff23f3eb478a817a50e3de287ba437ce5b3" exitCode=0
Nov 01 12:44:07.479969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:07.479605    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g" event={"ID":"0c576d1c-8cab-4947-b614-c4159156994e","Type":"ContainerDied","Data":"201730e3268c927734d7e5a12eb9aff23f3eb478a817a50e3de287ba437ce5b3"}
Nov 01 12:44:08.049028 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.048993    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"]
Nov 01 12:44:08.056869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.056832    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b66d97e-lnrfx"]
Nov 01 12:44:08.622303 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.622267    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9115e0-22b8-443f-a831-5953676a38d3" path="/var/lib/kubelet/pods/ad9115e0-22b8-443f-a831-5953676a38d3/volumes"
Nov 01 12:44:08.693422 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.693395    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:08.866389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.866292    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-xks-ca\") pod \"0c576d1c-8cab-4947-b614-c4159156994e\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") "
Nov 01 12:44:08.866552 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.866389    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-openshift-ca\") pod \"0c576d1c-8cab-4947-b614-c4159156994e\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") "
Nov 01 12:44:08.866620 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.866585    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c576d1c-8cab-4947-b614-c4159156994e-kube-api-access-fx4bc\" (UniqueName: \"kubernetes.io/projected/0c576d1c-8cab-4947-b614-c4159156994e-kube-api-access-fx4bc\") pod \"0c576d1c-8cab-4947-b614-c4159156994e\" (UID: \"0c576d1c-8cab-4947-b614-c4159156994e\") "
Nov 01 12:44:08.866901 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.866872    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-openshift-ca" pod "0c576d1c-8cab-4947-b614-c4159156994e" (UID: "0c576d1c-8cab-4947-b614-c4159156994e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:08.867301 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.867283    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:08.869028 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.869001    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-xks-ca" pod "0c576d1c-8cab-4947-b614-c4159156994e" (UID: "0c576d1c-8cab-4947-b614-c4159156994e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:08.869302 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.869275    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c576d1c-8cab-4947-b614-c4159156994e-kube-api-access-fx4bc" pod "0c576d1c-8cab-4947-b614-c4159156994e" (UID: "0c576d1c-8cab-4947-b614-c4159156994e"). InnerVolumeSpecName "kube-api-access-fx4bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:08.969201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.969166    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx4bc\" (UniqueName: \"kubernetes.io/projected/0c576d1c-8cab-4947-b614-c4159156994e-kube-api-access-fx4bc\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:08.969201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:08.969196    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c576d1c-8cab-4947-b614-c4159156994e-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:09.499118 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:09.499092    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"
Nov 01 12:44:09.499398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:09.499115    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g" event={"ID":"0c576d1c-8cab-4947-b614-c4159156994e","Type":"ContainerDied","Data":"1158bf048427a98a14ab2e29865611051818bee8172454e7a655f1a228502544"}
Nov 01 12:44:09.499398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:09.499163    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1158bf048427a98a14ab2e29865611051818bee8172454e7a655f1a228502544"
Nov 01 12:44:10.863582 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.863542    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"]
Nov 01 12:44:10.864570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.864549    5725 cpu_manager.go:414] "containerMap: removing container" podUID="0c576d1c-8cab-4947-b614-c4159156994e" containerName="revoke-keys"
Nov 01 12:44:10.864641 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.864573    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c576d1c-8cab-4947-b614-c4159156994e" containerName="revoke-keys"
Nov 01 12:44:10.864793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.864781    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c576d1c-8cab-4947-b614-c4159156994e" containerName="revoke-keys"
Nov 01 12:44:10.869705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.869683    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:10.900145 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.900115    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2s9x\" (UniqueName: \"kubernetes.io/projected/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-kube-api-access-f2s9x\") pod \"maas-api-revoke-sub-9774a4a8-ln7q6\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:10.900306 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.900274    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9774a4a8-ln7q6\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:10.900473 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.900457    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9774a4a8-ln7q6\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:10.922097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.920365    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"]
Nov 01 12:44:10.960292 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.960259    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:44:10.960490 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:10.960334    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:44:11.002480 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:11.002440    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9774a4a8-ln7q6\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:11.002947 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:11.002920    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9774a4a8-ln7q6\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:11.003189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:11.003165    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2s9x\" (UniqueName: \"kubernetes.io/projected/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-kube-api-access-f2s9x\") pod \"maas-api-revoke-sub-9774a4a8-ln7q6\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:11.003763 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:11.003730    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9774a4a8-ln7q6\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:11.006390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:11.006368    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9774a4a8-ln7q6\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:11.057758 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:11.057725    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2s9x\" (UniqueName: \"kubernetes.io/projected/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-kube-api-access-f2s9x\") pod \"maas-api-revoke-sub-9774a4a8-ln7q6\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:11.185211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:11.185129    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:11.579277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:11.579242    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"]
Nov 01 12:44:11.583287 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:11.583260    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7cecc5_5b8c_4d88_9b9e_220ee75730c0.slice/crio-fbcf1835601c3a073587cf5ed2e815cd92947e608501bc47aecd0e29b0a8b0d9 WatchSource:0}: Error finding container fbcf1835601c3a073587cf5ed2e815cd92947e608501bc47aecd0e29b0a8b0d9: Status 404 returned error can't find the container with id fbcf1835601c3a073587cf5ed2e815cd92947e608501bc47aecd0e29b0a8b0d9
Nov 01 12:44:12.529340 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:12.529308    5725 generic.go:356] "Generic (PLEG): container finished" podID="0c7cecc5-5b8c-4d88-9b9e-220ee75730c0" containerID="5ebaf3abc1c1f7016c70c32be4c0f1ca76eaff779b6d0bf6f601e78e52be217a" exitCode=0
Nov 01 12:44:12.529768 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:12.529393    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6" event={"ID":"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0","Type":"ContainerDied","Data":"5ebaf3abc1c1f7016c70c32be4c0f1ca76eaff779b6d0bf6f601e78e52be217a"}
Nov 01 12:44:12.529768 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:12.529460    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6" event={"ID":"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0","Type":"ContainerStarted","Data":"fbcf1835601c3a073587cf5ed2e815cd92947e608501bc47aecd0e29b0a8b0d9"}
Nov 01 12:44:14.252110 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:14.241286    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.321076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:14.317940    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.358077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:14.326881    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.391074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:14.390670    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.536345 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:14.536217    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.796189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:14.770133    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.234150 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.224202    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.342125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.338119    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.360695 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.360652    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.420265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.394432    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.545665 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.545191    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:44:15.599643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.599606    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:44:15.600711 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.600680    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:44:15.676420 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.676390    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:44:15.676699 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.676679    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:44:15.694076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.652769    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Nov 01 12:44:15.694372 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.675640    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Nov 01 12:44:15.851133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.825470    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.853792 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:15.853764    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:44:16.211075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.210260    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:16.233088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.229281    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Nov 01 12:44:16.233088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.230145    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:44:16.392520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.383141    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:16.494084 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.430195    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:44:16.521074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.494687    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-xks-ca\") pod \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") "
Nov 01 12:44:16.521074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.494827    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-kube-api-access-f2s9x\" (UniqueName: \"kubernetes.io/projected/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-kube-api-access-f2s9x\") pod \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") "
Nov 01 12:44:16.521074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.494904    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-openshift-ca\") pod \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\" (UID: \"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0\") "
Nov 01 12:44:16.521074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.498627    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-openshift-ca" pod "0c7cecc5-5b8c-4d88-9b9e-220ee75730c0" (UID: "0c7cecc5-5b8c-4d88-9b9e-220ee75730c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:16.553593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.532319    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:16.553593 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.550260    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:16.603075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.599388    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:16.627277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.627214    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-kube-api-access-f2s9x" pod "0c7cecc5-5b8c-4d88-9b9e-220ee75730c0" (UID: "0c7cecc5-5b8c-4d88-9b9e-220ee75730c0"). InnerVolumeSpecName "kube-api-access-f2s9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:16.710128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.703217    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-xks-ca" pod "0c7cecc5-5b8c-4d88-9b9e-220ee75730c0" (UID: "0c7cecc5-5b8c-4d88-9b9e-220ee75730c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:16.775078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.774381    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:16.775078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.774421    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2s9x\" (UniqueName: \"kubernetes.io/projected/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0-kube-api-access-f2s9x\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:16.906132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:16.889724    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:44:17.140080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:17.122173    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:17.140080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:17.122241    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:17.247204 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:17.247155    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:17.398762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:17.387182    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:44:17.398762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:17.387990    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:17.450076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:17.438641    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:17.540076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:17.536147    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:44:17.586276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:17.560556    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.089076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:18.072980    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6" event={"ID":"0c7cecc5-5b8c-4d88-9b9e-220ee75730c0","Type":"ContainerDied","Data":"fbcf1835601c3a073587cf5ed2e815cd92947e608501bc47aecd0e29b0a8b0d9"}
Nov 01 12:44:18.089076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:18.073101    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbcf1835601c3a073587cf5ed2e815cd92947e608501bc47aecd0e29b0a8b0d9"
Nov 01 12:44:18.089076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:18.073394    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"
Nov 01 12:44:18.115334 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:18.115274    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:44:18.274335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:18.274288    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.433830 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:18.433718    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.601813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:18.600564    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:44:18.682223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:18.682171    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.749181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:18.734032    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:19.141131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:19.115930    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:19.312693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:19.283712    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:19.469124 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:19.435028    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"]
Nov 01 12:44:19.727158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:19.714227    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e688374-wpjbk"]
Nov 01 12:44:19.727158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:19.714275    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"]
Nov 01 12:44:19.727158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:19.714325    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:44:19.727158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:19.450257    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:19.731068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:19.640159    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:19.740888 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:19.740847    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:20.078888 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.078145    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823a85ef-j6585"]
Nov 01 12:44:20.235306 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.235260    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"]
Nov 01 12:44:20.281582 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.281548    5725 cpu_manager.go:414] "containerMap: removing container" podUID="0c7cecc5-5b8c-4d88-9b9e-220ee75730c0" containerName="revoke-keys"
Nov 01 12:44:20.281840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.281820    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7cecc5-5b8c-4d88-9b9e-220ee75730c0" containerName="revoke-keys"
Nov 01 12:44:20.282273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.282253    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7cecc5-5b8c-4d88-9b9e-220ee75730c0" containerName="revoke-keys"
Nov 01 12:44:20.392638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.392551    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.399395 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.399350    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:20.409080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.399954    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.733946    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.734023    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.734096    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.734139    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.734180    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.734241    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.734302    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.734368    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.734433    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg5nl\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-kube-api-access-rg5nl\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.735653    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.740180    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.776489    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:44:20.854642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.802369    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:20.932633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.932590    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.932923 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.932900    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.933125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.933104    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.933297 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.933278    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.933439 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.933423    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.933597 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.933580    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.933768 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.933753    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.933929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.933912    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.934140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.934104    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg5nl\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-kube-api-access-rg5nl\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.935767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.935732    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.947073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.941114    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.947073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.941393    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.947073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.941639    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:20.950078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:20.949372    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:21.124123 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.121512    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11363d9f-3a67-4427-abaf-c32e25968810" path="/var/lib/kubelet/pods/11363d9f-3a67-4427-abaf-c32e25968810/volumes"
Nov 01 12:44:21.124123 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.122268    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616b7f3c-0d6e-4f3d-8df2-594665010978" path="/var/lib/kubelet/pods/616b7f3c-0d6e-4f3d-8df2-594665010978/volumes"
Nov 01 12:44:21.124123 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.122729    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"]
Nov 01 12:44:21.138604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.138570    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg5nl\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-kube-api-access-rg5nl\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:21.177849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.171675    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:21.209741 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.203413    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:21.210838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.210537    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:21.263540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.263512    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:44:21.352075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.343618    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:21.473932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.450974    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:21.718823 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.714241    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:21.743405 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.741149    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:21.743405 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.743032    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:21.787094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.784122    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:44:21.809667 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:21.808752    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:22.101162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.096298    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:44:22.105071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.105028    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:44:22.111005 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.110975    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:44:22.111255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.111224    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:44:22.129554 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.126021    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:44:22.416939 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.415947    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"]
Nov 01 12:44:22.423276 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:22.423250    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93cbe0a_0e18_4aae_b2dc_aeae691aeb3d.slice/crio-e995aa8a5d844a240a6ba367e7b634516ac6929ea67c17ec162cb8e4b7c8976a WatchSource:0}: Error finding container e995aa8a5d844a240a6ba367e7b634516ac6929ea67c17ec162cb8e4b7c8976a: Status 404 returned error can't find the container with id e995aa8a5d844a240a6ba367e7b634516ac6929ea67c17ec162cb8e4b7c8976a
Nov 01 12:44:22.426262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.426214    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:44:22.426347 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.426312    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:44:22.426347 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.426342    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:44:22.969429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.969388    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"]
Nov 01 12:44:22.975308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:22.975279    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.024436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.024402    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"]
Nov 01 12:44:23.024898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.024871    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:44:23.025064 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.025025    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:44:23.025410 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.025388    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2791dc32-6jtdg\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.025498 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.025429    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2791dc32-6jtdg\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.025498 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.025485    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fsl\" (UniqueName: \"kubernetes.io/projected/97d4279a-f0fe-4658-a98f-b8143c1d37df-kube-api-access-k9fsl\") pod \"maas-api-revoke-sub-2791dc32-6jtdg\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.114093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.114014    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"]
Nov 01 12:44:23.118868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.118843    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.126896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.126866    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2791dc32-6jtdg\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.127022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.126925    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2791dc32-6jtdg\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.127022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.126963    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fsl\" (UniqueName: \"kubernetes.io/projected/97d4279a-f0fe-4658-a98f-b8143c1d37df-kube-api-access-k9fsl\") pod \"maas-api-revoke-sub-2791dc32-6jtdg\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.127775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.127739    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2791dc32-6jtdg\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.129878 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.129852    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2791dc32-6jtdg\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.136440 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.136403    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" event={"ID":"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d","Type":"ContainerStarted","Data":"0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae"}
Nov 01 12:44:23.136565 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.136459    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" event={"ID":"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d","Type":"ContainerStarted","Data":"e995aa8a5d844a240a6ba367e7b634516ac6929ea67c17ec162cb8e4b7c8976a"}
Nov 01 12:44:23.172615 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.172585    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"]
Nov 01 12:44:23.183119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.182672    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fsl\" (UniqueName: \"kubernetes.io/projected/97d4279a-f0fe-4658-a98f-b8143c1d37df-kube-api-access-k9fsl\") pod \"maas-api-revoke-sub-2791dc32-6jtdg\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.235761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.235715    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7bt\" (UniqueName: \"kubernetes.io/projected/be4b9e99-2af5-465d-a539-3e410025a08f-kube-api-access-js7bt\") pod \"maas-api-revoke-sub-f0bdb7eb-fz7nj\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.236159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.236133    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0bdb7eb-fz7nj\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.236860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.236841    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0bdb7eb-fz7nj\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.290461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.290432    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:23.374109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.343572    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0bdb7eb-fz7nj\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.374109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.343719    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7bt\" (UniqueName: \"kubernetes.io/projected/be4b9e99-2af5-465d-a539-3e410025a08f-kube-api-access-js7bt\") pod \"maas-api-revoke-sub-f0bdb7eb-fz7nj\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.374109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.343801    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0bdb7eb-fz7nj\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.374109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.344697    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0bdb7eb-fz7nj\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.374109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.345811    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" podStartSLOduration=5.345795269 podStartE2EDuration="5.345795269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:23.342808787 +0000 UTC m=+1304.850000003" watchObservedRunningTime="2026-11-01 12:44:23.345795269 +0000 UTC m=+1304.852986487"
Nov 01 12:44:23.374109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.364516    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0bdb7eb-fz7nj\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.374109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.364586    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:23.374609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.374295    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 12:44:23.374609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.374351    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 12:44:23.421620 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.404904    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7bt\" (UniqueName: \"kubernetes.io/projected/be4b9e99-2af5-465d-a539-3e410025a08f-kube-api-access-js7bt\") pod \"maas-api-revoke-sub-f0bdb7eb-fz7nj\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.461083 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.446278    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:23.725079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.724934    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"]
Nov 01 12:44:23.776808 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:23.776763    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d4279a_f0fe_4658_a98f_b8143c1d37df.slice/crio-73e69bf50e043d3f8f07e81f5e444b9b0155e4d9488ce25a53e5e14c1fb1090c WatchSource:0}: Error finding container 73e69bf50e043d3f8f07e81f5e444b9b0155e4d9488ce25a53e5e14c1fb1090c: Status 404 returned error can't find the container with id 73e69bf50e043d3f8f07e81f5e444b9b0155e4d9488ce25a53e5e14c1fb1090c
Nov 01 12:44:23.893603 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:23.893563    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"]
Nov 01 12:44:23.945345 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:23.941164    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4b9e99_2af5_465d_a539_3e410025a08f.slice/crio-94f2fcc01944a15e4b8d9407b8baba150815604e7a03197a89251446aad6b6c0 WatchSource:0}: Error finding container 94f2fcc01944a15e4b8d9407b8baba150815604e7a03197a89251446aad6b6c0: Status 404 returned error can't find the container with id 94f2fcc01944a15e4b8d9407b8baba150815604e7a03197a89251446aad6b6c0
Nov 01 12:44:24.180063 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:24.180019    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg" event={"ID":"97d4279a-f0fe-4658-a98f-b8143c1d37df","Type":"ContainerStarted","Data":"73e69bf50e043d3f8f07e81f5e444b9b0155e4d9488ce25a53e5e14c1fb1090c"}
Nov 01 12:44:24.181976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:24.181952    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj" event={"ID":"be4b9e99-2af5-465d-a539-3e410025a08f","Type":"ContainerStarted","Data":"94f2fcc01944a15e4b8d9407b8baba150815604e7a03197a89251446aad6b6c0"}
Nov 01 12:44:24.363320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:24.359527    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 12:44:24.363320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:24.359575    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 12:44:25.063184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:25.063103    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"]
Nov 01 12:44:25.069058 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:25.069012    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0468bc6e-2kdt8"]
Nov 01 12:44:25.192771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:25.192688    5725 generic.go:356] "Generic (PLEG): container finished" podID="be4b9e99-2af5-465d-a539-3e410025a08f" containerID="9d8a9110b82934041f67af76e4eae22af9655465896a1e9b8f47410e191a38da" exitCode=0
Nov 01 12:44:25.193175 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:25.192767    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj" event={"ID":"be4b9e99-2af5-465d-a539-3e410025a08f","Type":"ContainerDied","Data":"9d8a9110b82934041f67af76e4eae22af9655465896a1e9b8f47410e191a38da"}
Nov 01 12:44:25.194846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:25.194816    5725 generic.go:356] "Generic (PLEG): container finished" podID="97d4279a-f0fe-4658-a98f-b8143c1d37df" containerID="397b53b96b2b4edf75291bf30cf5a67666fe0ef71e0a070040635e5fcd6eb76a" exitCode=0
Nov 01 12:44:25.194991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:25.194896    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg" event={"ID":"97d4279a-f0fe-4658-a98f-b8143c1d37df","Type":"ContainerDied","Data":"397b53b96b2b4edf75291bf30cf5a67666fe0ef71e0a070040635e5fcd6eb76a"}
Nov 01 12:44:25.369073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:25.359486    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 12:44:25.369073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:25.359545    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 12:44:26.067101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.067059    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"]
Nov 01 12:44:26.074962 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.074932    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d88c85-7drzz"]
Nov 01 12:44:26.384969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.372845    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 12:44:26.384969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.372901    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 12:44:26.552100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.548984    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:26.552100 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.551410    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:26.618954 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.618923    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d68db58-a145-4f99-98d5-9a3df9919c4f" path="/var/lib/kubelet/pods/3d68db58-a145-4f99-98d5-9a3df9919c4f/volumes"
Nov 01 12:44:26.619481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.619463    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985b82b0-c281-4f06-9b8c-12956a950dd9" path="/var/lib/kubelet/pods/985b82b0-c281-4f06-9b8c-12956a950dd9/volumes"
Nov 01 12:44:26.624727 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.624705    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-openshift-ca\") pod \"be4b9e99-2af5-465d-a539-3e410025a08f\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") "
Nov 01 12:44:26.624853 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.624839    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d4279a-f0fe-4658-a98f-b8143c1d37df-kube-api-access-k9fsl\" (UniqueName: \"kubernetes.io/projected/97d4279a-f0fe-4658-a98f-b8143c1d37df-kube-api-access-k9fsl\") pod \"97d4279a-f0fe-4658-a98f-b8143c1d37df\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") "
Nov 01 12:44:26.624918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.624903    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4b9e99-2af5-465d-a539-3e410025a08f-kube-api-access-js7bt\" (UniqueName: \"kubernetes.io/projected/be4b9e99-2af5-465d-a539-3e410025a08f-kube-api-access-js7bt\") pod \"be4b9e99-2af5-465d-a539-3e410025a08f\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") "
Nov 01 12:44:26.624978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.624949    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-xks-ca\") pod \"be4b9e99-2af5-465d-a539-3e410025a08f\" (UID: \"be4b9e99-2af5-465d-a539-3e410025a08f\") "
Nov 01 12:44:26.625037 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.624996    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-xks-ca\") pod \"97d4279a-f0fe-4658-a98f-b8143c1d37df\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") "
Nov 01 12:44:26.625113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.625063    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-openshift-ca\") pod \"97d4279a-f0fe-4658-a98f-b8143c1d37df\" (UID: \"97d4279a-f0fe-4658-a98f-b8143c1d37df\") "
Nov 01 12:44:26.625761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.625706    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-openshift-ca" pod "be4b9e99-2af5-465d-a539-3e410025a08f" (UID: "be4b9e99-2af5-465d-a539-3e410025a08f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:26.626193 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.626123    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-openshift-ca" pod "97d4279a-f0fe-4658-a98f-b8143c1d37df" (UID: "97d4279a-f0fe-4658-a98f-b8143c1d37df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:26.656116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.626855    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:26.656116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.626881    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:26.656116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.640918    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d4279a-f0fe-4658-a98f-b8143c1d37df-kube-api-access-k9fsl" pod "97d4279a-f0fe-4658-a98f-b8143c1d37df" (UID: "97d4279a-f0fe-4658-a98f-b8143c1d37df"). InnerVolumeSpecName "kube-api-access-k9fsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:26.656116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.645769    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-xks-ca" pod "97d4279a-f0fe-4658-a98f-b8143c1d37df" (UID: "97d4279a-f0fe-4658-a98f-b8143c1d37df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:26.656116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.645856    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-xks-ca" pod "be4b9e99-2af5-465d-a539-3e410025a08f" (UID: "be4b9e99-2af5-465d-a539-3e410025a08f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:26.695247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.693152    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4b9e99-2af5-465d-a539-3e410025a08f-kube-api-access-js7bt" pod "be4b9e99-2af5-465d-a539-3e410025a08f" (UID: "be4b9e99-2af5-465d-a539-3e410025a08f"). InnerVolumeSpecName "kube-api-access-js7bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:26.743070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.742821    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be4b9e99-2af5-465d-a539-3e410025a08f-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:26.743070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.742861    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d4279a-f0fe-4658-a98f-b8143c1d37df-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:26.743070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.742878    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9fsl\" (UniqueName: \"kubernetes.io/projected/97d4279a-f0fe-4658-a98f-b8143c1d37df-kube-api-access-k9fsl\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:26.743070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:26.742896    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7bt\" (UniqueName: \"kubernetes.io/projected/be4b9e99-2af5-465d-a539-3e410025a08f-kube-api-access-js7bt\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:27.215347 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.215312    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg" event={"ID":"97d4279a-f0fe-4658-a98f-b8143c1d37df","Type":"ContainerDied","Data":"73e69bf50e043d3f8f07e81f5e444b9b0155e4d9488ce25a53e5e14c1fb1090c"}
Nov 01 12:44:27.215522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.215398    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e69bf50e043d3f8f07e81f5e444b9b0155e4d9488ce25a53e5e14c1fb1090c"
Nov 01 12:44:27.215522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.215465    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"
Nov 01 12:44:27.219203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.219168    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj" event={"ID":"be4b9e99-2af5-465d-a539-3e410025a08f","Type":"ContainerDied","Data":"94f2fcc01944a15e4b8d9407b8baba150815604e7a03197a89251446aad6b6c0"}
Nov 01 12:44:27.219337 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.219219    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f2fcc01944a15e4b8d9407b8baba150815604e7a03197a89251446aad6b6c0"
Nov 01 12:44:27.219337 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.219256    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"
Nov 01 12:44:27.279784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.279745    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"]
Nov 01 12:44:27.280474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.280458    5725 cpu_manager.go:414] "containerMap: removing container" podUID="be4b9e99-2af5-465d-a539-3e410025a08f" containerName="revoke-keys"
Nov 01 12:44:27.280536 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.280475    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4b9e99-2af5-465d-a539-3e410025a08f" containerName="revoke-keys"
Nov 01 12:44:27.280536 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.280501    5725 cpu_manager.go:414] "containerMap: removing container" podUID="97d4279a-f0fe-4658-a98f-b8143c1d37df" containerName="revoke-keys"
Nov 01 12:44:27.280536 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.280507    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d4279a-f0fe-4658-a98f-b8143c1d37df" containerName="revoke-keys"
Nov 01 12:44:27.280760 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.280603    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d4279a-f0fe-4658-a98f-b8143c1d37df" containerName="revoke-keys"
Nov 01 12:44:27.280760 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.280753    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4b9e99-2af5-465d-a539-3e410025a08f" containerName="revoke-keys"
Nov 01 12:44:27.284852 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.284833    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.368723 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.368689    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:44:27.369147 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.369130    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:44:27.370702 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.370679    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 12:44:27.370892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.370838    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 12:44:27.373535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.373510    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"]
Nov 01 12:44:27.381103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.380544    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df28901e-4f5k7\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.381103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.380628    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jf24\" (UniqueName: \"kubernetes.io/projected/235e00ca-7d40-49eb-bfde-2b15163e5ab1-kube-api-access-2jf24\") pod \"maas-api-revoke-sub-df28901e-4f5k7\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.381103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.380707    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df28901e-4f5k7\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.484510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.484465    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df28901e-4f5k7\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.484965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.484672    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df28901e-4f5k7\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.484965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.484781    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jf24\" (UniqueName: \"kubernetes.io/projected/235e00ca-7d40-49eb-bfde-2b15163e5ab1-kube-api-access-2jf24\") pod \"maas-api-revoke-sub-df28901e-4f5k7\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.485824 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.485794    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df28901e-4f5k7\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.489591 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.489374    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df28901e-4f5k7\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.574328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.574187    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jf24\" (UniqueName: \"kubernetes.io/projected/235e00ca-7d40-49eb-bfde-2b15163e5ab1-kube-api-access-2jf24\") pod \"maas-api-revoke-sub-df28901e-4f5k7\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:27.599064 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:27.599022    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:28.062074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.061682    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"]
Nov 01 12:44:28.079774 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:28.079537    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235e00ca_7d40_49eb_bfde_2b15163e5ab1.slice/crio-b712aa96e900e0b3d00775278dd610d240f59b2f6d1f5bde45ce1d26991732f0 WatchSource:0}: Error finding container b712aa96e900e0b3d00775278dd610d240f59b2f6d1f5bde45ce1d26991732f0: Status 404 returned error can't find the container with id b712aa96e900e0b3d00775278dd610d240f59b2f6d1f5bde45ce1d26991732f0
Nov 01 12:44:28.219557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.219518    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"]
Nov 01 12:44:28.226232 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.226195    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.260586 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.260542    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7" event={"ID":"235e00ca-7d40-49eb-bfde-2b15163e5ab1","Type":"ContainerStarted","Data":"b712aa96e900e0b3d00775278dd610d240f59b2f6d1f5bde45ce1d26991732f0"}
Nov 01 12:44:28.306217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.306175    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aea6b6a-8wnpb\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.306695 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.306670    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aea6b6a-8wnpb\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.306855 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.306841    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5kcn\" (UniqueName: \"kubernetes.io/projected/d7186b20-4ae3-4cdd-90de-09282817dd58-kube-api-access-t5kcn\") pod \"maas-api-revoke-sub-4aea6b6a-8wnpb\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.313513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.313433    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"]
Nov 01 12:44:28.358075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.357490    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 12:44:28.358075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.357569    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 12:44:28.408592 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.408559    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aea6b6a-8wnpb\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.408807 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.408794    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5kcn\" (UniqueName: \"kubernetes.io/projected/d7186b20-4ae3-4cdd-90de-09282817dd58-kube-api-access-t5kcn\") pod \"maas-api-revoke-sub-4aea6b6a-8wnpb\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.408941 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.408929    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aea6b6a-8wnpb\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.409329 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.409308    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aea6b6a-8wnpb\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.411410 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.411388    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aea6b6a-8wnpb\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.469070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.468097    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5kcn\" (UniqueName: \"kubernetes.io/projected/d7186b20-4ae3-4cdd-90de-09282817dd58-kube-api-access-t5kcn\") pod \"maas-api-revoke-sub-4aea6b6a-8wnpb\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:28.609533 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:28.609502    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:29.043181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:29.043146    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"]
Nov 01 12:44:29.048530 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:29.048496    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7186b20_4ae3_4cdd_90de_09282817dd58.slice/crio-a6f464338226c6f01cdfc8bd5b0af26e4ce8ebccf72dd301802c66820f59eb30 WatchSource:0}: Error finding container a6f464338226c6f01cdfc8bd5b0af26e4ce8ebccf72dd301802c66820f59eb30: Status 404 returned error can't find the container with id a6f464338226c6f01cdfc8bd5b0af26e4ce8ebccf72dd301802c66820f59eb30
Nov 01 12:44:29.274812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:29.274667    5725 generic.go:356] "Generic (PLEG): container finished" podID="235e00ca-7d40-49eb-bfde-2b15163e5ab1" containerID="f90738678ffcd69b05fffcd82b544c160a206b6d0e27bec4a2f93a47d29f4ae4" exitCode=0
Nov 01 12:44:29.274812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:29.274786    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7" event={"ID":"235e00ca-7d40-49eb-bfde-2b15163e5ab1","Type":"ContainerDied","Data":"f90738678ffcd69b05fffcd82b544c160a206b6d0e27bec4a2f93a47d29f4ae4"}
Nov 01 12:44:29.276665 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:29.276636    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb" event={"ID":"d7186b20-4ae3-4cdd-90de-09282817dd58","Type":"ContainerStarted","Data":"a6f464338226c6f01cdfc8bd5b0af26e4ce8ebccf72dd301802c66820f59eb30"}
Nov 01 12:44:29.353863 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:29.353836    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 12:44:29.354020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:29.353881    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 12:44:30.286654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.286620    5725 generic.go:356] "Generic (PLEG): container finished" podID="d7186b20-4ae3-4cdd-90de-09282817dd58" containerID="f13466baca856d0616be8c9760e02bba87c2d8fad8184b18491b005b765be936" exitCode=0
Nov 01 12:44:30.287038 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.286708    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb" event={"ID":"d7186b20-4ae3-4cdd-90de-09282817dd58","Type":"ContainerDied","Data":"f13466baca856d0616be8c9760e02bba87c2d8fad8184b18491b005b765be936"}
Nov 01 12:44:30.361269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.361235    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:30.362157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.362087    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:30.363489 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.363463    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:44:30.480061 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.480027    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:30.551816 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.551705    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-openshift-ca\") pod \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") "
Nov 01 12:44:30.552152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.552076    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-xks-ca\") pod \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") "
Nov 01 12:44:30.552270 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.552153    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235e00ca-7d40-49eb-bfde-2b15163e5ab1-kube-api-access-2jf24\" (UniqueName: \"kubernetes.io/projected/235e00ca-7d40-49eb-bfde-2b15163e5ab1-kube-api-access-2jf24\") pod \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\" (UID: \"235e00ca-7d40-49eb-bfde-2b15163e5ab1\") "
Nov 01 12:44:30.552965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.552940    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-openshift-ca" pod "235e00ca-7d40-49eb-bfde-2b15163e5ab1" (UID: "235e00ca-7d40-49eb-bfde-2b15163e5ab1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:30.557436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.557385    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235e00ca-7d40-49eb-bfde-2b15163e5ab1-kube-api-access-2jf24" pod "235e00ca-7d40-49eb-bfde-2b15163e5ab1" (UID: "235e00ca-7d40-49eb-bfde-2b15163e5ab1"). InnerVolumeSpecName "kube-api-access-2jf24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:30.557719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.557701    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-xks-ca" pod "235e00ca-7d40-49eb-bfde-2b15163e5ab1" (UID: "235e00ca-7d40-49eb-bfde-2b15163e5ab1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:30.653805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.653771    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:30.653805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.653798    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jf24\" (UniqueName: \"kubernetes.io/projected/235e00ca-7d40-49eb-bfde-2b15163e5ab1-kube-api-access-2jf24\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:30.653805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:30.653810    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235e00ca-7d40-49eb-bfde-2b15163e5ab1-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:31.031571 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:44:31.031527    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:44:31.299039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.298948    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"
Nov 01 12:44:31.299039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.298981    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7" event={"ID":"235e00ca-7d40-49eb-bfde-2b15163e5ab1","Type":"ContainerDied","Data":"b712aa96e900e0b3d00775278dd610d240f59b2f6d1f5bde45ce1d26991732f0"}
Nov 01 12:44:31.299039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.299065    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b712aa96e900e0b3d00775278dd610d240f59b2f6d1f5bde45ce1d26991732f0"
Nov 01 12:44:31.469116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.469072    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"]
Nov 01 12:44:31.469374 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.469315    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy" containerID="cri-o://e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742" gracePeriod=30
Nov 01 12:44:31.513027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.512999    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:31.514509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.514487    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"]
Nov 01 12:44:31.514676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.514655    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy" containerID="cri-o://1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85" gracePeriod=30
Nov 01 12:44:31.570789 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.570705    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"]
Nov 01 12:44:31.571479 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.571444    5725 cpu_manager.go:414] "containerMap: removing container" podUID="235e00ca-7d40-49eb-bfde-2b15163e5ab1" containerName="revoke-keys"
Nov 01 12:44:31.571479 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.571478    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235e00ca-7d40-49eb-bfde-2b15163e5ab1" containerName="revoke-keys"
Nov 01 12:44:31.571657 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.571490    5725 cpu_manager.go:414] "containerMap: removing container" podUID="d7186b20-4ae3-4cdd-90de-09282817dd58" containerName="revoke-keys"
Nov 01 12:44:31.571657 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.571496    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7186b20-4ae3-4cdd-90de-09282817dd58" containerName="revoke-keys"
Nov 01 12:44:31.571657 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.571640    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="235e00ca-7d40-49eb-bfde-2b15163e5ab1" containerName="revoke-keys"
Nov 01 12:44:31.571657 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.571648    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7186b20-4ae3-4cdd-90de-09282817dd58" containerName="revoke-keys"
Nov 01 12:44:31.575542 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.575521    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.673109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.672094    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-openshift-ca\") pod \"d7186b20-4ae3-4cdd-90de-09282817dd58\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") "
Nov 01 12:44:31.673109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.672484    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-openshift-ca" pod "d7186b20-4ae3-4cdd-90de-09282817dd58" (UID: "d7186b20-4ae3-4cdd-90de-09282817dd58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:31.673109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.672650    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7186b20-4ae3-4cdd-90de-09282817dd58-kube-api-access-t5kcn\" (UniqueName: \"kubernetes.io/projected/d7186b20-4ae3-4cdd-90de-09282817dd58-kube-api-access-t5kcn\") pod \"d7186b20-4ae3-4cdd-90de-09282817dd58\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") "
Nov 01 12:44:31.673109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.672751    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-xks-ca\") pod \"d7186b20-4ae3-4cdd-90de-09282817dd58\" (UID: \"d7186b20-4ae3-4cdd-90de-09282817dd58\") "
Nov 01 12:44:31.676173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676141    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.676315 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676284    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5qs\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-kube-api-access-6g5qs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.676389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676377    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.676445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676430    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.676505 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676467    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.680219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676544    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.680219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676599    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.680219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676646    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.680219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676705    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.680219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.676962    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-xks-ca" pod "d7186b20-4ae3-4cdd-90de-09282817dd58" (UID: "d7186b20-4ae3-4cdd-90de-09282817dd58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:31.680219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.677268    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:31.680219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.677317    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7186b20-4ae3-4cdd-90de-09282817dd58-kube-api-access-t5kcn" pod "d7186b20-4ae3-4cdd-90de-09282817dd58" (UID: "d7186b20-4ae3-4cdd-90de-09282817dd58"). InnerVolumeSpecName "kube-api-access-t5kcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:31.680219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.677468    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"]
Nov 01 12:44:31.778487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.778444    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5qs\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-kube-api-access-6g5qs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.778667 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.778514    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.778667 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.778571    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.778667 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.778609    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.779063 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.778703    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.779063 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.778895    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.779063 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.778980    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.779217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.779064    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.779285 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.779262    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.780487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.779363    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.780487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.779394    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.780487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.780333    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.780487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.780383    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5kcn\" (UniqueName: \"kubernetes.io/projected/d7186b20-4ae3-4cdd-90de-09282817dd58-kube-api-access-t5kcn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:31.780487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.780404    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7186b20-4ae3-4cdd-90de-09282817dd58-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:31.780741 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.780627    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.781306 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.781282    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.782188 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.782087    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.782681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.782660    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.832805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.832724    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5qs\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-kube-api-access-6g5qs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.832805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.832763    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:31.896239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:31.896206    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:32.042739 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.042707    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"]
Nov 01 12:44:32.048968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.048944    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.099584 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.099322    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"]
Nov 01 12:44:32.193304 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.193274    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5h2k\" (UniqueName: \"kubernetes.io/projected/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-kube-api-access-r5h2k\") pod \"maas-api-revoke-sub-0e76cbf6-725c6\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.193474 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.193320    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e76cbf6-725c6\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.193527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.193472    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e76cbf6-725c6\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.294838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.294800    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5h2k\" (UniqueName: \"kubernetes.io/projected/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-kube-api-access-r5h2k\") pod \"maas-api-revoke-sub-0e76cbf6-725c6\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.295009 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.294863    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e76cbf6-725c6\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.295009 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.294990    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e76cbf6-725c6\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.295686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.295655    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e76cbf6-725c6\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.297230 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.297207    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e76cbf6-725c6\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.309781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.309756    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"
Nov 01 12:44:32.310162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.309780    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb" event={"ID":"d7186b20-4ae3-4cdd-90de-09282817dd58","Type":"ContainerDied","Data":"a6f464338226c6f01cdfc8bd5b0af26e4ce8ebccf72dd301802c66820f59eb30"}
Nov 01 12:44:32.310162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.309827    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f464338226c6f01cdfc8bd5b0af26e4ce8ebccf72dd301802c66820f59eb30"
Nov 01 12:44:32.346028 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.345999    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"]
Nov 01 12:44:32.347216 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:32.347191    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56ceeda_6a87_4ebb_b482_df5f1bae1706.slice/crio-3fabe9b7162bd17189714573dd9c0247050bd768502751e41983a73774ffc805 WatchSource:0}: Error finding container 3fabe9b7162bd17189714573dd9c0247050bd768502751e41983a73774ffc805: Status 404 returned error can't find the container with id 3fabe9b7162bd17189714573dd9c0247050bd768502751e41983a73774ffc805
Nov 01 12:44:32.349084 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.349010    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5h2k\" (UniqueName: \"kubernetes.io/projected/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-kube-api-access-r5h2k\") pod \"maas-api-revoke-sub-0e76cbf6-725c6\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.350020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.349967    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:44:32.350112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.350029    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:44:32.350152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.350105    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:44:32.367650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.367625    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:32.780804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:32.780745    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"]
Nov 01 12:44:32.803129 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:32.788851    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2f5a6d_f878_45aa_81c6_ac638a7e9c5d.slice/crio-2b35e11b7871bcf4e68935c65b41ad6be4302fe093c66c22680198034757a2b3 WatchSource:0}: Error finding container 2b35e11b7871bcf4e68935c65b41ad6be4302fe093c66c22680198034757a2b3: Status 404 returned error can't find the container with id 2b35e11b7871bcf4e68935c65b41ad6be4302fe093c66c22680198034757a2b3
Nov 01 12:44:33.319972 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.319872    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6" event={"ID":"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d","Type":"ContainerStarted","Data":"921c8342be234870c6dc811775ed8a05f8ef8eb6bd4e220e8e8ba38e2d7db8a1"}
Nov 01 12:44:33.319972 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.319955    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6" event={"ID":"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d","Type":"ContainerStarted","Data":"2b35e11b7871bcf4e68935c65b41ad6be4302fe093c66c22680198034757a2b3"}
Nov 01 12:44:33.321995 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.321965    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" event={"ID":"b56ceeda-6a87-4ebb-b482-df5f1bae1706","Type":"ContainerStarted","Data":"ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc"}
Nov 01 12:44:33.322161 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.322020    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" event={"ID":"b56ceeda-6a87-4ebb-b482-df5f1bae1706","Type":"ContainerStarted","Data":"3fabe9b7162bd17189714573dd9c0247050bd768502751e41983a73774ffc805"}
Nov 01 12:44:33.443074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.439022    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6" podStartSLOduration=2.439004539 podStartE2EDuration="2.439004539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:33.43842729 +0000 UTC m=+1314.945618507" watchObservedRunningTime="2026-11-01 12:44:33.439004539 +0000 UTC m=+1314.946195758"
Nov 01 12:44:33.569323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.569245    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" podStartSLOduration=2.569220153 podStartE2EDuration="2.569220153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:33.555375158 +0000 UTC m=+1315.062566391" watchObservedRunningTime="2026-11-01 12:44:33.569220153 +0000 UTC m=+1315.076411374"
Nov 01 12:44:33.897354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.897314    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:33.898862 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.898835    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 12:44:33.899000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.898883    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 12:44:33.971295 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.971253    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"]
Nov 01 12:44:33.976127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:33.976103    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.042717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.037149    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"]
Nov 01 12:44:34.071210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.071165    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"]
Nov 01 12:44:34.079200 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.079157    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"]
Nov 01 12:44:34.094707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.088255    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124629a1-kx82v"]
Nov 01 12:44:34.097982 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.097945    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74bee244-bzhdq"]
Nov 01 12:44:34.125118 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.125040    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89633d72-74e0-437d-819b-97f5f4143306-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248038df-nlnr4\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.125282 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.125261    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88zvg\" (UniqueName: \"kubernetes.io/projected/89633d72-74e0-437d-819b-97f5f4143306-kube-api-access-88zvg\") pod \"maas-api-revoke-sub-248038df-nlnr4\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.125359 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.125333    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89633d72-74e0-437d-819b-97f5f4143306-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248038df-nlnr4\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.227004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.226968    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88zvg\" (UniqueName: \"kubernetes.io/projected/89633d72-74e0-437d-819b-97f5f4143306-kube-api-access-88zvg\") pod \"maas-api-revoke-sub-248038df-nlnr4\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.227221 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.227083    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89633d72-74e0-437d-819b-97f5f4143306-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248038df-nlnr4\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.227291 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.227248    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89633d72-74e0-437d-819b-97f5f4143306-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248038df-nlnr4\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.250136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.228155    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89633d72-74e0-437d-819b-97f5f4143306-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248038df-nlnr4\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.250136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.248416    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89633d72-74e0-437d-819b-97f5f4143306-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248038df-nlnr4\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.281575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.281534    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88zvg\" (UniqueName: \"kubernetes.io/projected/89633d72-74e0-437d-819b-97f5f4143306-kube-api-access-88zvg\") pod \"maas-api-revoke-sub-248038df-nlnr4\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.352078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.351362    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:34.358385 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.358354    5725 generic.go:356] "Generic (PLEG): container finished" podID="3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d" containerID="921c8342be234870c6dc811775ed8a05f8ef8eb6bd4e220e8e8ba38e2d7db8a1" exitCode=0
Nov 01 12:44:34.359201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.359176    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6" event={"ID":"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d","Type":"ContainerDied","Data":"921c8342be234870c6dc811775ed8a05f8ef8eb6bd4e220e8e8ba38e2d7db8a1"}
Nov 01 12:44:34.618328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.618243    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="437039d8-3fa5-4b26-b207-145ca139a447" path="/var/lib/kubelet/pods/437039d8-3fa5-4b26-b207-145ca139a447/volumes"
Nov 01 12:44:34.618805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.618790    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9ec72c-dcb2-4e39-8d8e-014287745a36" path="/var/lib/kubelet/pods/7e9ec72c-dcb2-4e39-8d8e-014287745a36/volumes"
Nov 01 12:44:34.759402 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:34.759370    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89633d72_74e0_437d_819b_97f5f4143306.slice/crio-b5c8e96f8ff86a8b05ff7cc2a27854175273dd284bd6ec0b2c435f8d5c28786e WatchSource:0}: Error finding container b5c8e96f8ff86a8b05ff7cc2a27854175273dd284bd6ec0b2c435f8d5c28786e: Status 404 returned error can't find the container with id b5c8e96f8ff86a8b05ff7cc2a27854175273dd284bd6ec0b2c435f8d5c28786e
Nov 01 12:44:34.760575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.760547    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"]
Nov 01 12:44:34.897993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.897919    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 12:44:34.897993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:34.897976    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 12:44:35.057142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.057043    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"]
Nov 01 12:44:35.062154 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.062124    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c99dbc5-d7hlk"]
Nov 01 12:44:35.371681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.371404    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4" event={"ID":"89633d72-74e0-437d-819b-97f5f4143306","Type":"ContainerStarted","Data":"a03deeb01a14c746c96356ab29a5d5a7056655e8365c533f6234a067d3ff178a"}
Nov 01 12:44:35.371681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.371473    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4" event={"ID":"89633d72-74e0-437d-819b-97f5f4143306","Type":"ContainerStarted","Data":"b5c8e96f8ff86a8b05ff7cc2a27854175273dd284bd6ec0b2c435f8d5c28786e"}
Nov 01 12:44:35.486078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.484627    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4" podStartSLOduration=2.484606265 podStartE2EDuration="2.484606265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:35.480819633 +0000 UTC m=+1316.988010874" watchObservedRunningTime="2026-11-01 12:44:35.484606265 +0000 UTC m=+1316.991797483"
Nov 01 12:44:35.674633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.674603    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:35.700900 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.700865    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-xks-ca\") pod \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") "
Nov 01 12:44:35.701115 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.700941    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-kube-api-access-r5h2k\" (UniqueName: \"kubernetes.io/projected/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-kube-api-access-r5h2k\") pod \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") "
Nov 01 12:44:35.701303 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.701282    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-openshift-ca\") pod \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\" (UID: \"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d\") "
Nov 01 12:44:35.701801 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.701778    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-openshift-ca" pod "3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d" (UID: "3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:35.702187 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.702168    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:35.705365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.703750    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-xks-ca" pod "3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d" (UID: "3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:35.705365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.704318    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-kube-api-access-r5h2k" pod "3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d" (UID: "3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d"). InnerVolumeSpecName "kube-api-access-r5h2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:35.804262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.804226    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:35.804262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.804259    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5h2k\" (UniqueName: \"kubernetes.io/projected/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d-kube-api-access-r5h2k\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:35.896817 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.896785    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 12:44:35.896962 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:35.896837    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 12:44:36.383116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:36.383083    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"
Nov 01 12:44:36.383538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:36.383139    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6" event={"ID":"3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d","Type":"ContainerDied","Data":"2b35e11b7871bcf4e68935c65b41ad6be4302fe093c66c22680198034757a2b3"}
Nov 01 12:44:36.383538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:36.383213    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b35e11b7871bcf4e68935c65b41ad6be4302fe093c66c22680198034757a2b3"
Nov 01 12:44:36.384915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:36.384892    5725 generic.go:356] "Generic (PLEG): container finished" podID="89633d72-74e0-437d-819b-97f5f4143306" containerID="a03deeb01a14c746c96356ab29a5d5a7056655e8365c533f6234a067d3ff178a" exitCode=0
Nov 01 12:44:36.385087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:36.384976    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4" event={"ID":"89633d72-74e0-437d-819b-97f5f4143306","Type":"ContainerDied","Data":"a03deeb01a14c746c96356ab29a5d5a7056655e8365c533f6234a067d3ff178a"}
Nov 01 12:44:36.619373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:36.619340    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179f214c-6ed2-40dd-bda9-51841f2125a9" path="/var/lib/kubelet/pods/179f214c-6ed2-40dd-bda9-51841f2125a9/volumes"
Nov 01 12:44:36.901860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:36.901340    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 12:44:36.901860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:36.901414    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 12:44:36.908126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:36.907851    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:44:37.025108 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.023123    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:44:37.026668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.026635    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-podinfo\") pod \"7b69a426-38af-4e4a-a378-86100a1a8aac\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") "
Nov 01 12:44:37.026789 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.026751    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-socket\") pod \"7b69a426-38af-4e4a-a378-86100a1a8aac\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") "
Nov 01 12:44:37.026843 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.026789    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-envoy\") pod \"7b69a426-38af-4e4a-a378-86100a1a8aac\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") "
Nov 01 12:44:37.026907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.026855    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-certs\") pod \"7b69a426-38af-4e4a-a378-86100a1a8aac\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") "
Nov 01 12:44:37.045165 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.043401    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-certs" pod "7b69a426-38af-4e4a-a378-86100a1a8aac" (UID: "7b69a426-38af-4e4a-a378-86100a1a8aac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.045647 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.045619    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-socket" pod "7b69a426-38af-4e4a-a378-86100a1a8aac" (UID: "7b69a426-38af-4e4a-a378-86100a1a8aac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.048154    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-data\") pod \"7b69a426-38af-4e4a-a378-86100a1a8aac\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") "
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.048406    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-data" pod "7b69a426-38af-4e4a-a378-86100a1a8aac" (UID: "7b69a426-38af-4e4a-a378-86100a1a8aac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.048497    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-credential-socket\") pod \"7b69a426-38af-4e4a-a378-86100a1a8aac\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") "
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.048541    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-kube-api-access-jtv6s\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-kube-api-access-jtv6s\") pod \"7b69a426-38af-4e4a-a378-86100a1a8aac\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") "
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.048616    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-token\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-token\") pod \"7b69a426-38af-4e4a-a378-86100a1a8aac\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") "
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.048671    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b69a426-38af-4e4a-a378-86100a1a8aac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69a426-38af-4e4a-a378-86100a1a8aac-istiod-ca-cert\") pod \"7b69a426-38af-4e4a-a378-86100a1a8aac\" (UID: \"7b69a426-38af-4e4a-a378-86100a1a8aac\") "
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.049404    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.049424    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.049438    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.057238    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b69a426-38af-4e4a-a378-86100a1a8aac-istiod-ca-cert" pod "7b69a426-38af-4e4a-a378-86100a1a8aac" (UID: "7b69a426-38af-4e4a-a378-86100a1a8aac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:37.059113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.057510    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-credential-socket" pod "7b69a426-38af-4e4a-a378-86100a1a8aac" (UID: "7b69a426-38af-4e4a-a378-86100a1a8aac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.075682 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.075643    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-podinfo" pod "7b69a426-38af-4e4a-a378-86100a1a8aac" (UID: "7b69a426-38af-4e4a-a378-86100a1a8aac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:44:37.075847 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.075685    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-envoy" pod "7b69a426-38af-4e4a-a378-86100a1a8aac" (UID: "7b69a426-38af-4e4a-a378-86100a1a8aac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.077140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.077098    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-kube-api-access-jtv6s" pod "7b69a426-38af-4e4a-a378-86100a1a8aac" (UID: "7b69a426-38af-4e4a-a378-86100a1a8aac"). InnerVolumeSpecName "kube-api-access-jtv6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:37.077649 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.077614    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-token" pod "7b69a426-38af-4e4a-a378-86100a1a8aac" (UID: "7b69a426-38af-4e4a-a378-86100a1a8aac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.162543    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-podinfo\") pod \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") "
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.162623    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-token\") pod \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") "
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.162665    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istiod-ca-cert\") pod \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") "
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.163232    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-socket\") pod \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") "
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.163276    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-envoy\") pod \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") "
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.163318    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-certs\") pod \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") "
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.163371    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-data\") pod \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") "
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.163504    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-kube-api-access-knqz4\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-kube-api-access-knqz4\") pod \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") "
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.163672    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istiod-ca-cert" pod "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" (UID: "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.163691    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-credential-socket\") pod \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\" (UID: \"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b\") "
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.163966    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-credential-socket" pod "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" (UID: "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164161    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-socket" pod "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" (UID: "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164452    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-certs" pod "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" (UID: "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164808    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164827    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164845    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164895    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164909    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69a426-38af-4e4a-a378-86100a1a8aac-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164922    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtv6s\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-kube-api-access-jtv6s\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164925    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-data" pod "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" (UID: "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164936    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164950    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69a426-38af-4e4a-a378-86100a1a8aac-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164965    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69a426-38af-4e4a-a378-86100a1a8aac-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.165245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.164978    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.168714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.166629    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-podinfo" pod "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" (UID: "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:44:37.168714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.167424    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-envoy" pod "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" (UID: "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:44:37.170508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.170485    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-kube-api-access-knqz4" pod "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" (UID: "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b"). InnerVolumeSpecName "kube-api-access-knqz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:37.171152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.171117    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-token" pod "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" (UID: "3b0b4044-3bc3-4ede-a6a1-e5d4973db18b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:37.265952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.265855    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.265952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.265887    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.265952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.265898    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knqz4\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-kube-api-access-knqz4\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.265952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.265910    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.265952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.265918    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.373568 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.373514    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"]
Nov 01 12:44:37.374667 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.374611    5725 cpu_manager.go:414] "containerMap: removing container" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy"
Nov 01 12:44:37.374742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.374672    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy"
Nov 01 12:44:37.374742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.374701    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d" containerName="revoke-keys"
Nov 01 12:44:37.374742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.374706    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d" containerName="revoke-keys"
Nov 01 12:44:37.374742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.374715    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy"
Nov 01 12:44:37.374742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.374722    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy"
Nov 01 12:44:37.374881 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.374859    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerName="istio-proxy"
Nov 01 12:44:37.374881 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.374868    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d" containerName="revoke-keys"
Nov 01 12:44:37.374881 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.374875    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerName="istio-proxy"
Nov 01 12:44:37.380728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.380314    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.398501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.398468    5725 generic.go:356] "Generic (PLEG): container finished" podID="7b69a426-38af-4e4a-a378-86100a1a8aac" containerID="e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742" exitCode=0
Nov 01 12:44:37.398943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.398539    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" event={"ID":"7b69a426-38af-4e4a-a378-86100a1a8aac","Type":"ContainerDied","Data":"e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742"}
Nov 01 12:44:37.399153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.399131    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd" event={"ID":"7b69a426-38af-4e4a-a378-86100a1a8aac","Type":"ContainerDied","Data":"ace9539b13f24afc178373e5f658ea7a4e7061aff1146fc43cdc898b53c6c968"}
Nov 01 12:44:37.399288 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.399274    5725 scope.go:122] "RemoveContainer" containerID="e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742"
Nov 01 12:44:37.399436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.398573    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"
Nov 01 12:44:37.405317 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.405283    5725 generic.go:356] "Generic (PLEG): container finished" podID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" containerID="1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85" exitCode=0
Nov 01 12:44:37.405392 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.405346    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"
Nov 01 12:44:37.405392 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.405354    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" event={"ID":"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b","Type":"ContainerDied","Data":"1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85"}
Nov 01 12:44:37.405498 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.405402    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq" event={"ID":"3b0b4044-3bc3-4ede-a6a1-e5d4973db18b","Type":"ContainerDied","Data":"f7960dfb6e1e0921f4f34f5cc31e89ac105debfcf4b956565336af2738acf508"}
Nov 01 12:44:37.424996 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.424963    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"]
Nov 01 12:44:37.467165 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.464747    5725 scope.go:122] "RemoveContainer" containerID="e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742"
Nov 01 12:44:37.467165 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:44:37.466116    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742\": container with ID starting with e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742 not found: ID does not exist" containerID="e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742"
Nov 01 12:44:37.467165 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.466152    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742"} err="failed to get container status \"e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742\": rpc error: code = NotFound desc = could not find container \"e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742\": container with ID starting with e1778ec878689ec1ef7bba81fa664991bc471e38179aa58fdc0272a0c7853742 not found: ID does not exist"
Nov 01 12:44:37.467165 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.466173    5725 scope.go:122] "RemoveContainer" containerID="1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85"
Nov 01 12:44:37.471664 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.471615    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfabce44-m75x7\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.471812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.471779    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql48g\" (UniqueName: \"kubernetes.io/projected/0477fd2f-de1f-4860-86dd-4489b71fceaa-kube-api-access-ql48g\") pod \"maas-api-revoke-sub-cfabce44-m75x7\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.471889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.471870    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfabce44-m75x7\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.489014 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.488982    5725 scope.go:122] "RemoveContainer" containerID="1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85"
Nov 01 12:44:37.489653 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:44:37.489620    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85\": container with ID starting with 1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85 not found: ID does not exist" containerID="1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85"
Nov 01 12:44:37.489787 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.489661    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85"} err="failed to get container status \"1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85\": rpc error: code = NotFound desc = could not find container \"1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85\": container with ID starting with 1963a9f3d8effdaf7e82676e670d902de7194ff8aeaad94415767c5bbe109f85 not found: ID does not exist"
Nov 01 12:44:37.575233 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.575190    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfabce44-m75x7\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.575424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.575312    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql48g\" (UniqueName: \"kubernetes.io/projected/0477fd2f-de1f-4860-86dd-4489b71fceaa-kube-api-access-ql48g\") pod \"maas-api-revoke-sub-cfabce44-m75x7\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.575424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.575379    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfabce44-m75x7\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.576345 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.576318    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfabce44-m75x7\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.578122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.578104    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfabce44-m75x7\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.683903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.683798    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql48g\" (UniqueName: \"kubernetes.io/projected/0477fd2f-de1f-4860-86dd-4489b71fceaa-kube-api-access-ql48g\") pod \"maas-api-revoke-sub-cfabce44-m75x7\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.685664 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.685606    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"]
Nov 01 12:44:37.691216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.691190    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-vw7gq"]
Nov 01 12:44:37.695656 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.695634    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:37.776348 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.776320    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:37.786532 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.786500    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"]
Nov 01 12:44:37.793876 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.793841    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-tpczd"]
Nov 01 12:44:37.885030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.884322    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89633d72-74e0-437d-819b-97f5f4143306-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89633d72-74e0-437d-819b-97f5f4143306-maas-api-xks-ca\") pod \"89633d72-74e0-437d-819b-97f5f4143306\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") "
Nov 01 12:44:37.885030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.884400    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89633d72-74e0-437d-819b-97f5f4143306-kube-api-access-88zvg\" (UniqueName: \"kubernetes.io/projected/89633d72-74e0-437d-819b-97f5f4143306-kube-api-access-88zvg\") pod \"89633d72-74e0-437d-819b-97f5f4143306\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") "
Nov 01 12:44:37.885030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.884464    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89633d72-74e0-437d-819b-97f5f4143306-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89633d72-74e0-437d-819b-97f5f4143306-maas-api-openshift-ca\") pod \"89633d72-74e0-437d-819b-97f5f4143306\" (UID: \"89633d72-74e0-437d-819b-97f5f4143306\") "
Nov 01 12:44:37.886414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.886378    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89633d72-74e0-437d-819b-97f5f4143306-maas-api-openshift-ca" pod "89633d72-74e0-437d-819b-97f5f4143306" (UID: "89633d72-74e0-437d-819b-97f5f4143306"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:37.893932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.887581    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89633d72-74e0-437d-819b-97f5f4143306-maas-api-xks-ca" pod "89633d72-74e0-437d-819b-97f5f4143306" (UID: "89633d72-74e0-437d-819b-97f5f4143306"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:37.893932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.889528    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89633d72-74e0-437d-819b-97f5f4143306-kube-api-access-88zvg" pod "89633d72-74e0-437d-819b-97f5f4143306" (UID: "89633d72-74e0-437d-819b-97f5f4143306"). InnerVolumeSpecName "kube-api-access-88zvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:37.898410 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.898373    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 12:44:37.898534 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.898433    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 12:44:37.987631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.987596    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89633d72-74e0-437d-819b-97f5f4143306-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.987631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.987626    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88zvg\" (UniqueName: \"kubernetes.io/projected/89633d72-74e0-437d-819b-97f5f4143306-kube-api-access-88zvg\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:37.987631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:37.987637    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89633d72-74e0-437d-819b-97f5f4143306-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:38.161553 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:38.161515    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0477fd2f_de1f_4860_86dd_4489b71fceaa.slice/crio-f597ff7b07a0ceeb8af391ff8f55bf95d03f34542839318ab80166e02d3dc978 WatchSource:0}: Error finding container f597ff7b07a0ceeb8af391ff8f55bf95d03f34542839318ab80166e02d3dc978: Status 404 returned error can't find the container with id f597ff7b07a0ceeb8af391ff8f55bf95d03f34542839318ab80166e02d3dc978
Nov 01 12:44:38.161963 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.161939    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"]
Nov 01 12:44:38.416926 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.416890    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4" event={"ID":"89633d72-74e0-437d-819b-97f5f4143306","Type":"ContainerDied","Data":"b5c8e96f8ff86a8b05ff7cc2a27854175273dd284bd6ec0b2c435f8d5c28786e"}
Nov 01 12:44:38.416926 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.416911    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"
Nov 01 12:44:38.417400 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.416980    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5c8e96f8ff86a8b05ff7cc2a27854175273dd284bd6ec0b2c435f8d5c28786e"
Nov 01 12:44:38.419866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.419834    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7" event={"ID":"0477fd2f-de1f-4860-86dd-4489b71fceaa","Type":"ContainerStarted","Data":"f597ff7b07a0ceeb8af391ff8f55bf95d03f34542839318ab80166e02d3dc978"}
Nov 01 12:44:38.618860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.618775    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0b4044-3bc3-4ede-a6a1-e5d4973db18b" path="/var/lib/kubelet/pods/3b0b4044-3bc3-4ede-a6a1-e5d4973db18b/volumes"
Nov 01 12:44:38.619368 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.619344    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b69a426-38af-4e4a-a378-86100a1a8aac" path="/var/lib/kubelet/pods/7b69a426-38af-4e4a-a378-86100a1a8aac/volumes"
Nov 01 12:44:38.896813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.896727    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Nov 01 12:44:38.896813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.896790    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Nov 01 12:44:38.906898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.906863    5725 scope.go:122] "RemoveContainer" containerID="e2019802688b028584dc0b2158be6331ce4c971b588cdc00c03831a54a60cee9"
Nov 01 12:44:38.918578 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.918557    5725 scope.go:122] "RemoveContainer" containerID="8fd851b3e6fe57c8c87ee038e8c5870f5c8d6935148d77e542d5d1ff2398b4dd"
Nov 01 12:44:38.930526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.930487    5725 scope.go:122] "RemoveContainer" containerID="7451c010d7b64e96653c1916af018bd887803e085e93322a2e585e7c4aa03b60"
Nov 01 12:44:38.943463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.943433    5725 scope.go:122] "RemoveContainer" containerID="7a0b72c34a4d8d07787947236edd634fade0a176ee6bedfe04b79518b03dd07f"
Nov 01 12:44:38.954237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:38.954201    5725 scope.go:122] "RemoveContainer" containerID="b24e0a6d1fedc095d5e502b6459c15b14e9540f0f294a9a9475dc742530215f7"
Nov 01 12:44:39.431842 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:39.431801    5725 generic.go:356] "Generic (PLEG): container finished" podID="0477fd2f-de1f-4860-86dd-4489b71fceaa" containerID="aab1105e2f77195093ccb0810bde4e689d031b889a64c74870351bdb3be47eca" exitCode=0
Nov 01 12:44:39.432296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:39.431886    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7" event={"ID":"0477fd2f-de1f-4860-86dd-4489b71fceaa","Type":"ContainerDied","Data":"aab1105e2f77195093ccb0810bde4e689d031b889a64c74870351bdb3be47eca"}
Nov 01 12:44:39.998469 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:39.998440    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:39.998796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:39.998761    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:39.999670 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:39.999651    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:44:40.047481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.047447    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"]
Nov 01 12:44:40.094515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.094479    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b133beba-ghrpk"]
Nov 01 12:44:40.620362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.620332    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4561a347-386c-43a4-8163-65e47df1fd34" path="/var/lib/kubelet/pods/4561a347-386c-43a4-8163-65e47df1fd34/volumes"
Nov 01 12:44:40.640508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.640480    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:40.738167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.738131    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-xks-ca\") pod \"0477fd2f-de1f-4860-86dd-4489b71fceaa\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") "
Nov 01 12:44:40.738388 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.738374    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-openshift-ca\") pod \"0477fd2f-de1f-4860-86dd-4489b71fceaa\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") "
Nov 01 12:44:40.738445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.738417    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0477fd2f-de1f-4860-86dd-4489b71fceaa-kube-api-access-ql48g\" (UniqueName: \"kubernetes.io/projected/0477fd2f-de1f-4860-86dd-4489b71fceaa-kube-api-access-ql48g\") pod \"0477fd2f-de1f-4860-86dd-4489b71fceaa\" (UID: \"0477fd2f-de1f-4860-86dd-4489b71fceaa\") "
Nov 01 12:44:40.738728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.738698    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-openshift-ca" pod "0477fd2f-de1f-4860-86dd-4489b71fceaa" (UID: "0477fd2f-de1f-4860-86dd-4489b71fceaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:40.738952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.738927    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:40.740395 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.740362    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-xks-ca" pod "0477fd2f-de1f-4860-86dd-4489b71fceaa" (UID: "0477fd2f-de1f-4860-86dd-4489b71fceaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:40.740480 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.740458    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0477fd2f-de1f-4860-86dd-4489b71fceaa-kube-api-access-ql48g" pod "0477fd2f-de1f-4860-86dd-4489b71fceaa" (UID: "0477fd2f-de1f-4860-86dd-4489b71fceaa"). InnerVolumeSpecName "kube-api-access-ql48g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:40.840418 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.840378    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql48g\" (UniqueName: \"kubernetes.io/projected/0477fd2f-de1f-4860-86dd-4489b71fceaa-kube-api-access-ql48g\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:40.840418 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:40.840415    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0477fd2f-de1f-4860-86dd-4489b71fceaa-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:41.454823 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:41.454792    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"
Nov 01 12:44:41.455017 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:41.454838    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7" event={"ID":"0477fd2f-de1f-4860-86dd-4489b71fceaa","Type":"ContainerDied","Data":"f597ff7b07a0ceeb8af391ff8f55bf95d03f34542839318ab80166e02d3dc978"}
Nov 01 12:44:41.455017 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:41.454898    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f597ff7b07a0ceeb8af391ff8f55bf95d03f34542839318ab80166e02d3dc978"
Nov 01 12:44:42.843486 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.843447    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"]
Nov 01 12:44:42.844520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.844499    5725 cpu_manager.go:414] "containerMap: removing container" podUID="0477fd2f-de1f-4860-86dd-4489b71fceaa" containerName="revoke-keys"
Nov 01 12:44:42.844520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.844523    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0477fd2f-de1f-4860-86dd-4489b71fceaa" containerName="revoke-keys"
Nov 01 12:44:42.844623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.844548    5725 cpu_manager.go:414] "containerMap: removing container" podUID="89633d72-74e0-437d-819b-97f5f4143306" containerName="revoke-keys"
Nov 01 12:44:42.844623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.844556    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89633d72-74e0-437d-819b-97f5f4143306" containerName="revoke-keys"
Nov 01 12:44:42.844776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.844763    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="0477fd2f-de1f-4860-86dd-4489b71fceaa" containerName="revoke-keys"
Nov 01 12:44:42.844811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.844781    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="89633d72-74e0-437d-819b-97f5f4143306" containerName="revoke-keys"
Nov 01 12:44:42.849570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.849551    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:42.897348 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.897313    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"]
Nov 01 12:44:42.937358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.937320    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:44:42.937568 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.937378    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:44:42.978117 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.978044    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfsw\" (UniqueName: \"kubernetes.io/projected/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-kube-api-access-2gfsw\") pod \"maas-api-revoke-sub-1805927e-b4hpk\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:42.978309 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.978192    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1805927e-b4hpk\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:42.978309 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:42.978225    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1805927e-b4hpk\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:43.079707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:43.079665    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfsw\" (UniqueName: \"kubernetes.io/projected/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-kube-api-access-2gfsw\") pod \"maas-api-revoke-sub-1805927e-b4hpk\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:43.079919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:43.079723    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1805927e-b4hpk\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:43.079919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:43.079743    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1805927e-b4hpk\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:43.080470 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:43.080443    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1805927e-b4hpk\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:43.082221 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:43.082200    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1805927e-b4hpk\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:43.135044 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:43.134967    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfsw\" (UniqueName: \"kubernetes.io/projected/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-kube-api-access-2gfsw\") pod \"maas-api-revoke-sub-1805927e-b4hpk\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:43.164832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:43.164797    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:43.562540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:43.562502    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"]
Nov 01 12:44:43.564385 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:43.564350    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0266bd6f_3bc0_45e4_b6d4_efcb2220776a.slice/crio-2fc75d3cb55db40b8afd201b0d494d043fd9b326176ac4ec48aaf292d26d4cd7 WatchSource:0}: Error finding container 2fc75d3cb55db40b8afd201b0d494d043fd9b326176ac4ec48aaf292d26d4cd7: Status 404 returned error can't find the container with id 2fc75d3cb55db40b8afd201b0d494d043fd9b326176ac4ec48aaf292d26d4cd7
Nov 01 12:44:44.082412 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.082375    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7091-4b54c1"]
Nov 01 12:44:44.087182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.087157    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-4b54c1"
Nov 01 12:44:44.142309 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.142276    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-4b54c1"]
Nov 01 12:44:44.182522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.182491    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rbvvl\""
Nov 01 12:44:44.195876 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.195816    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8445\" (UniqueName: \"kubernetes.io/projected/8210b8fe-b7fc-4330-abbf-70771e446e13-kube-api-access-p8445\") pod \"test-curl-7091-4b54c1\" (UID: \"8210b8fe-b7fc-4330-abbf-70771e446e13\") " pod="opendatahub/test-curl-7091-4b54c1"
Nov 01 12:44:44.297840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.297797    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8445\" (UniqueName: \"kubernetes.io/projected/8210b8fe-b7fc-4330-abbf-70771e446e13-kube-api-access-p8445\") pod \"test-curl-7091-4b54c1\" (UID: \"8210b8fe-b7fc-4330-abbf-70771e446e13\") " pod="opendatahub/test-curl-7091-4b54c1"
Nov 01 12:44:44.352380 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.352301    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8445\" (UniqueName: \"kubernetes.io/projected/8210b8fe-b7fc-4330-abbf-70771e446e13-kube-api-access-p8445\") pod \"test-curl-7091-4b54c1\" (UID: \"8210b8fe-b7fc-4330-abbf-70771e446e13\") " pod="opendatahub/test-curl-7091-4b54c1"
Nov 01 12:44:44.407131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.407098    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-4b54c1"
Nov 01 12:44:44.491819 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.491787    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk" event={"ID":"0266bd6f-3bc0-45e4-b6d4-efcb2220776a","Type":"ContainerStarted","Data":"42078165e4726257dd150f0983dcd38dacb7f5698a78df3b05e83f8269688e75"}
Nov 01 12:44:44.491980 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.491847    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk" event={"ID":"0266bd6f-3bc0-45e4-b6d4-efcb2220776a","Type":"ContainerStarted","Data":"2fc75d3cb55db40b8afd201b0d494d043fd9b326176ac4ec48aaf292d26d4cd7"}
Nov 01 12:44:44.607734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.607641    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk" podStartSLOduration=2.607626677 podStartE2EDuration="2.607626677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:44.605426662 +0000 UTC m=+1326.112617880" watchObservedRunningTime="2026-11-01 12:44:44.607626677 +0000 UTC m=+1326.114817893"
Nov 01 12:44:44.793105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.793072    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-4b54c1"]
Nov 01 12:44:44.797824 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:44.797793    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8210b8fe_b7fc_4330_abbf_70771e446e13.slice/crio-33b73d86667cab76a6cecfab7b15084de26dc3396b5315e9c0b7ecc7c556f9f8 WatchSource:0}: Error finding container 33b73d86667cab76a6cecfab7b15084de26dc3396b5315e9c0b7ecc7c556f9f8: Status 404 returned error can't find the container with id 33b73d86667cab76a6cecfab7b15084de26dc3396b5315e9c0b7ecc7c556f9f8
Nov 01 12:44:44.799966 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:44.799949    5725 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:44:45.048495 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:45.048458    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"]
Nov 01 12:44:45.053892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:45.053861    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe26baf1-zz855"]
Nov 01 12:44:45.506423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:45.506385    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-4b54c1" event={"ID":"8210b8fe-b7fc-4330-abbf-70771e446e13","Type":"ContainerStarted","Data":"33b73d86667cab76a6cecfab7b15084de26dc3396b5315e9c0b7ecc7c556f9f8"}
Nov 01 12:44:45.508189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:45.508162    5725 generic.go:356] "Generic (PLEG): container finished" podID="0266bd6f-3bc0-45e4-b6d4-efcb2220776a" containerID="42078165e4726257dd150f0983dcd38dacb7f5698a78df3b05e83f8269688e75" exitCode=0
Nov 01 12:44:45.508317 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:45.508248    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk" event={"ID":"0266bd6f-3bc0-45e4-b6d4-efcb2220776a","Type":"ContainerDied","Data":"42078165e4726257dd150f0983dcd38dacb7f5698a78df3b05e83f8269688e75"}
Nov 01 12:44:46.181284 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.181206    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"]
Nov 01 12:44:46.185941 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.185919    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.238371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.238340    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"]
Nov 01 12:44:46.326869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.326832    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e754e361-r5cnw\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.327041 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.326994    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdtp4\" (UniqueName: \"kubernetes.io/projected/6867e3c2-5bde-456d-a5d7-70640987a35f-kube-api-access-fdtp4\") pod \"maas-api-revoke-sub-e754e361-r5cnw\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.327252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.327230    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e754e361-r5cnw\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.428707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.428676    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e754e361-r5cnw\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.428885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.428745    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e754e361-r5cnw\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.428885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.428813    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdtp4\" (UniqueName: \"kubernetes.io/projected/6867e3c2-5bde-456d-a5d7-70640987a35f-kube-api-access-fdtp4\") pod \"maas-api-revoke-sub-e754e361-r5cnw\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.429606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.429582    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e754e361-r5cnw\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.439212 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.431489    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e754e361-r5cnw\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.483939 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.483907    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdtp4\" (UniqueName: \"kubernetes.io/projected/6867e3c2-5bde-456d-a5d7-70640987a35f-kube-api-access-fdtp4\") pod \"maas-api-revoke-sub-e754e361-r5cnw\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.501060 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.501020    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:46.521115 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.521080    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-4b54c1" event={"ID":"8210b8fe-b7fc-4330-abbf-70771e446e13","Type":"ContainerStarted","Data":"1fd466fa0559ecf6205f600006b34856744f624f2ad534ecba895a5d7541871f"}
Nov 01 12:44:46.623393 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.623355    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8449121a-e767-4186-a689-0a89692a6d81" path="/var/lib/kubelet/pods/8449121a-e767-4186-a689-0a89692a6d81/volumes"
Nov 01 12:44:46.633617 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.633567    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7091-4b54c1" podStartSLOduration=1.503158033 podStartE2EDuration="2.633553047s" totalImagesPullingTime="1.130395014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:46.629265433 +0000 UTC m=+1328.136456650" watchObservedRunningTime="2026-11-01 12:44:46.633553047 +0000 UTC m=+1328.140744264"
Nov 01 12:44:46.747291 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.747258    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:46.835862 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.835812    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-xks-ca\") pod \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") "
Nov 01 12:44:46.836041 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.835879    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-kube-api-access-2gfsw\" (UniqueName: \"kubernetes.io/projected/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-kube-api-access-2gfsw\") pod \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") "
Nov 01 12:44:46.836134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.836036    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-openshift-ca\") pod \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\" (UID: \"0266bd6f-3bc0-45e4-b6d4-efcb2220776a\") "
Nov 01 12:44:46.836427 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.836385    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-openshift-ca" pod "0266bd6f-3bc0-45e4-b6d4-efcb2220776a" (UID: "0266bd6f-3bc0-45e4-b6d4-efcb2220776a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:46.836797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.836778    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:46.838726 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.838700    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-xks-ca" pod "0266bd6f-3bc0-45e4-b6d4-efcb2220776a" (UID: "0266bd6f-3bc0-45e4-b6d4-efcb2220776a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:46.838726 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.838703    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-kube-api-access-2gfsw" pod "0266bd6f-3bc0-45e4-b6d4-efcb2220776a" (UID: "0266bd6f-3bc0-45e4-b6d4-efcb2220776a"). InnerVolumeSpecName "kube-api-access-2gfsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:46.895583 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.895546    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"]
Nov 01 12:44:46.938718 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:46.938677    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6867e3c2_5bde_456d_a5d7_70640987a35f.slice/crio-4ac93cc72c1fc8ab3f12b5d3fe1fd0eb4ed218447eb5490c841dcd8540df8da4 WatchSource:0}: Error finding container 4ac93cc72c1fc8ab3f12b5d3fe1fd0eb4ed218447eb5490c841dcd8540df8da4: Status 404 returned error can't find the container with id 4ac93cc72c1fc8ab3f12b5d3fe1fd0eb4ed218447eb5490c841dcd8540df8da4
Nov 01 12:44:46.938866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.938762    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:46.938866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:46.938789    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gfsw\" (UniqueName: \"kubernetes.io/projected/0266bd6f-3bc0-45e4-b6d4-efcb2220776a-kube-api-access-2gfsw\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:47.532440 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:47.532342    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw" event={"ID":"6867e3c2-5bde-456d-a5d7-70640987a35f","Type":"ContainerStarted","Data":"66917f7e8cfe097221d803fe9f234579de0f07037f524de23f7590cb7359fac8"}
Nov 01 12:44:47.532440 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:47.532398    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw" event={"ID":"6867e3c2-5bde-456d-a5d7-70640987a35f","Type":"ContainerStarted","Data":"4ac93cc72c1fc8ab3f12b5d3fe1fd0eb4ed218447eb5490c841dcd8540df8da4"}
Nov 01 12:44:47.534458 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:47.534433    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"
Nov 01 12:44:47.534458 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:47.534432    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk" event={"ID":"0266bd6f-3bc0-45e4-b6d4-efcb2220776a","Type":"ContainerDied","Data":"2fc75d3cb55db40b8afd201b0d494d043fd9b326176ac4ec48aaf292d26d4cd7"}
Nov 01 12:44:47.534654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:47.534495    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc75d3cb55db40b8afd201b0d494d043fd9b326176ac4ec48aaf292d26d4cd7"
Nov 01 12:44:47.646556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:47.646493    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw" podStartSLOduration=1.6464740789999999 podStartE2EDuration="1.646474079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:47.643288463 +0000 UTC m=+1329.150479703" watchObservedRunningTime="2026-11-01 12:44:47.646474079 +0000 UTC m=+1329.153665296"
Nov 01 12:44:48.039127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.039092    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"]
Nov 01 12:44:48.040204 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.040178    5725 cpu_manager.go:414] "containerMap: removing container" podUID="0266bd6f-3bc0-45e4-b6d4-efcb2220776a" containerName="revoke-keys"
Nov 01 12:44:48.040330 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.040205    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0266bd6f-3bc0-45e4-b6d4-efcb2220776a" containerName="revoke-keys"
Nov 01 12:44:48.040425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.040411    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="0266bd6f-3bc0-45e4-b6d4-efcb2220776a" containerName="revoke-keys"
Nov 01 12:44:48.045094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.045069    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.045255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.045228    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7091-4b54c1"]
Nov 01 12:44:48.050352 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.050327    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7091-4b54c1"]
Nov 01 12:44:48.053223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.053200    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c752746-cvl9x\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.053324 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.053258    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c752746-cvl9x\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.053595 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.053572    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkqnx\" (UniqueName: \"kubernetes.io/projected/f9c32452-c11e-4a58-ac55-e9841a6775b3-kube-api-access-xkqnx\") pod \"maas-api-revoke-sub-7c752746-cvl9x\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.106857 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.106821    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"]
Nov 01 12:44:48.154934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.154842    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkqnx\" (UniqueName: \"kubernetes.io/projected/f9c32452-c11e-4a58-ac55-e9841a6775b3-kube-api-access-xkqnx\") pod \"maas-api-revoke-sub-7c752746-cvl9x\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.155135 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.155002    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c752746-cvl9x\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.155135 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.155087    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c752746-cvl9x\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.155958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.155930    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c752746-cvl9x\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.158107 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.158083    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c752746-cvl9x\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.196109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.196067    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7091-8873ab"]
Nov 01 12:44:48.196857 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.196839    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8210b8fe-b7fc-4330-abbf-70771e446e13" containerName="test-curl-7091-4b54c1"
Nov 01 12:44:48.196918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.196857    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8210b8fe-b7fc-4330-abbf-70771e446e13" containerName="test-curl-7091-4b54c1"
Nov 01 12:44:48.196969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.196960    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8210b8fe-b7fc-4330-abbf-70771e446e13" containerName="test-curl-7091-4b54c1"
Nov 01 12:44:48.200977 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.200954    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:44:48.213317 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.210464    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkqnx\" (UniqueName: \"kubernetes.io/projected/f9c32452-c11e-4a58-ac55-e9841a6775b3-kube-api-access-xkqnx\") pod \"maas-api-revoke-sub-7c752746-cvl9x\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.254347 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.254309    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-8873ab"]
Nov 01 12:44:48.256729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.256694    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwncx\" (UniqueName: \"kubernetes.io/projected/785396d2-da8c-411f-bcde-58deae9061c9-kube-api-access-qwncx\") pod \"test-curl-7091-8873ab\" (UID: \"785396d2-da8c-411f-bcde-58deae9061c9\") " pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:44:48.358198 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.358150    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwncx\" (UniqueName: \"kubernetes.io/projected/785396d2-da8c-411f-bcde-58deae9061c9-kube-api-access-qwncx\") pod \"test-curl-7091-8873ab\" (UID: \"785396d2-da8c-411f-bcde-58deae9061c9\") " pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:44:48.364739 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.364696    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:48.415561 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.415475    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwncx\" (UniqueName: \"kubernetes.io/projected/785396d2-da8c-411f-bcde-58deae9061c9-kube-api-access-qwncx\") pod \"test-curl-7091-8873ab\" (UID: \"785396d2-da8c-411f-bcde-58deae9061c9\") " pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:44:48.529750 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.529715    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:44:48.549453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.549426    5725 generic.go:356] "Generic (PLEG): container finished" podID="6867e3c2-5bde-456d-a5d7-70640987a35f" containerID="66917f7e8cfe097221d803fe9f234579de0f07037f524de23f7590cb7359fac8" exitCode=0
Nov 01 12:44:48.549886 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.549519    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw" event={"ID":"6867e3c2-5bde-456d-a5d7-70640987a35f","Type":"ContainerDied","Data":"66917f7e8cfe097221d803fe9f234579de0f07037f524de23f7590cb7359fac8"}
Nov 01 12:44:48.549886 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.549803    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7091-4b54c1" podUID="8210b8fe-b7fc-4330-abbf-70771e446e13" containerName="test-curl-7091-4b54c1" containerID="cri-o://1fd466fa0559ecf6205f600006b34856744f624f2ad534ecba895a5d7541871f" gracePeriod=2
Nov 01 12:44:48.791466 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:48.791434    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"]
Nov 01 12:44:48.794719 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:44:48.794691    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c32452_c11e_4a58_ac55_e9841a6775b3.slice/crio-0ea4b997d944c1b40727d7dc33bd675de7b5989ca6d5c635a6e4def2cacfbb90 WatchSource:0}: Error finding container 0ea4b997d944c1b40727d7dc33bd675de7b5989ca6d5c635a6e4def2cacfbb90: Status 404 returned error can't find the container with id 0ea4b997d944c1b40727d7dc33bd675de7b5989ca6d5c635a6e4def2cacfbb90
Nov 01 12:44:49.174844 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:49.174636    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-8873ab"]
Nov 01 12:44:49.676076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:49.673546    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x" event={"ID":"f9c32452-c11e-4a58-ac55-e9841a6775b3","Type":"ContainerStarted","Data":"70a0f3a15cd74fd6a69856feb3ddf693953b26f8ee136e64f65c30c33966e8b3"}
Nov 01 12:44:49.676076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:49.673636    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x" event={"ID":"f9c32452-c11e-4a58-ac55-e9841a6775b3","Type":"ContainerStarted","Data":"0ea4b997d944c1b40727d7dc33bd675de7b5989ca6d5c635a6e4def2cacfbb90"}
Nov 01 12:44:49.707077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:49.706216    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-8873ab" event={"ID":"785396d2-da8c-411f-bcde-58deae9061c9","Type":"ContainerStarted","Data":"b2158fef763cb95dcc7d0fa83c4426f5e1ca4464866cb60a436e598122fe2614"}
Nov 01 12:44:49.707077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:49.706266    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-8873ab" event={"ID":"785396d2-da8c-411f-bcde-58deae9061c9","Type":"ContainerStarted","Data":"84f7c196503d833171da7ba75c664fa72e920fa2574dcbf543ca4a60c6dc97dd"}
Nov 01 12:44:49.973134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:49.965824    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7091-8873ab" podStartSLOduration=1.9657995160000001 podStartE2EDuration="1.965799516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:49.948581056 +0000 UTC m=+1331.455772290" watchObservedRunningTime="2026-11-01 12:44:49.965799516 +0000 UTC m=+1331.472990733"
Nov 01 12:44:49.973134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:49.965941    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x" podStartSLOduration=2.965930635 podStartE2EDuration="2.965930635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:49.8144358 +0000 UTC m=+1331.321627022" watchObservedRunningTime="2026-11-01 12:44:49.965930635 +0000 UTC m=+1331.473121852"
Nov 01 12:44:50.399139 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:50.393424    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:44:50.431919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:50.431528    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:44:50.567291 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:50.561555    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:50.755077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:50.752701    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:44:50.897090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:50.888355    5725 generic.go:356] "Generic (PLEG): container finished" podID="f9c32452-c11e-4a58-ac55-e9841a6775b3" containerID="70a0f3a15cd74fd6a69856feb3ddf693953b26f8ee136e64f65c30c33966e8b3" exitCode=0
Nov 01 12:44:50.897090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:50.889767    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x" event={"ID":"f9c32452-c11e-4a58-ac55-e9841a6775b3","Type":"ContainerDied","Data":"70a0f3a15cd74fd6a69856feb3ddf693953b26f8ee136e64f65c30c33966e8b3"}
Nov 01 12:44:51.182159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:50.952924    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.325744 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.284131    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.426095 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.405227    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.426095 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.406369    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:44:51.437390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.437209    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.482294 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.475391    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:44:51.611934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.593112    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.613277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.612913    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:44:51.646087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.633598    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:51.646087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.633664    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.718168 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.714385    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.758240 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.757142    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.809285 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.758835    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:44:51.813940 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.812718    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"]
Nov 01 12:44:51.829876 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.815690    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:51.894141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.894109    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159909d0-2ct68"]
Nov 01 12:44:51.960181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.959709    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.960782 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:51.960759    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:44:52.035176 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.035141    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-xks-ca\") pod \"6867e3c2-5bde-456d-a5d7-70640987a35f\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") "
Nov 01 12:44:52.065793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.065750    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-openshift-ca\") pod \"6867e3c2-5bde-456d-a5d7-70640987a35f\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") "
Nov 01 12:44:52.066190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.066166    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6867e3c2-5bde-456d-a5d7-70640987a35f-kube-api-access-fdtp4\" (UniqueName: \"kubernetes.io/projected/6867e3c2-5bde-456d-a5d7-70640987a35f-kube-api-access-fdtp4\") pod \"6867e3c2-5bde-456d-a5d7-70640987a35f\" (UID: \"6867e3c2-5bde-456d-a5d7-70640987a35f\") "
Nov 01 12:44:52.207364 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.207319    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6867e3c2-5bde-456d-a5d7-70640987a35f-kube-api-access-fdtp4" pod "6867e3c2-5bde-456d-a5d7-70640987a35f" (UID: "6867e3c2-5bde-456d-a5d7-70640987a35f"). InnerVolumeSpecName "kube-api-access-fdtp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:52.213085 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.103200    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-openshift-ca" pod "6867e3c2-5bde-456d-a5d7-70640987a35f" (UID: "6867e3c2-5bde-456d-a5d7-70640987a35f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:52.348609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.324171    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-xks-ca" pod "6867e3c2-5bde-456d-a5d7-70640987a35f" (UID: "6867e3c2-5bde-456d-a5d7-70640987a35f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:52.348609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.325136    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.348609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.327036    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:52.348609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.327085    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6867e3c2-5bde-456d-a5d7-70640987a35f-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:52.348609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.327101    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdtp4\" (UniqueName: \"kubernetes.io/projected/6867e3c2-5bde-456d-a5d7-70640987a35f-kube-api-access-fdtp4\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:52.363151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.354344    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.363151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.360943    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:44:52.436382 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.436343    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw" event={"ID":"6867e3c2-5bde-456d-a5d7-70640987a35f","Type":"ContainerDied","Data":"4ac93cc72c1fc8ab3f12b5d3fe1fd0eb4ed218447eb5490c841dcd8540df8da4"}
Nov 01 12:44:52.436654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.436634    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ac93cc72c1fc8ab3f12b5d3fe1fd0eb4ed218447eb5490c841dcd8540df8da4"
Nov 01 12:44:52.436843 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.436828    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"
Nov 01 12:44:52.476325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.476290    5725 generic.go:356] "Generic (PLEG): container finished" podID="8210b8fe-b7fc-4330-abbf-70771e446e13" containerID="1fd466fa0559ecf6205f600006b34856744f624f2ad534ecba895a5d7541871f" exitCode=137
Nov 01 12:44:52.478452 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.478420    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.563076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.500380    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.596547 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.596166    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.871142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.866840    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:44:52.977076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:52.960800    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:53.462075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:53.453456    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d288684f-59f6-42ba-a058-0404dfba1d12" path="/var/lib/kubelet/pods/d288684f-59f6-42ba-a058-0404dfba1d12/volumes"
Nov 01 12:44:53.549598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:53.549547    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:53.700909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:53.598417    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:44:53.700909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:53.683280    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:53.849077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:53.840910    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:53.849077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:53.840970    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:53.849077 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:44:53.841155    5725 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.214s"
Nov 01 12:44:53.875076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:53.872405    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.026076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.024495    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:54.026076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.024567    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.029182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.029121    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.291167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.290212    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:54.291167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.290294    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.434081 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.426733    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.724363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.724315    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6" podUID="40cb0bff-44e1-409a-b7fe-891015b30fcb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.67:50052\" within 1s: context deadline exceeded"
Nov 01 12:44:54.805241 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.805198    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.830511 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.810119    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.879077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.810187    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.880197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.880144    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:44:54.881289 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.881179    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-4b54c1"
Nov 01 12:44:54.957255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.957219    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:54.957573 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.957527    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.982287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.982198    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:54.982540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.982508    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.983884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:54.983855    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8210b8fe-b7fc-4330-abbf-70771e446e13-kube-api-access-p8445\" (UniqueName: \"kubernetes.io/projected/8210b8fe-b7fc-4330-abbf-70771e446e13-kube-api-access-p8445\") pod \"8210b8fe-b7fc-4330-abbf-70771e446e13\" (UID: \"8210b8fe-b7fc-4330-abbf-70771e446e13\") "
Nov 01 12:44:55.047109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.032676    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.047444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.047408    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Nov 01 12:44:55.083179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.083125    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:55.133310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.133261    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.133585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.110281    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:55.133797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.133771    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.442222 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.439566    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.588630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.588595    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:55.588831 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.588799    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.648129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.645695    5725 scope.go:122] "RemoveContainer" containerID="1fd466fa0559ecf6205f600006b34856744f624f2ad534ecba895a5d7541871f"
Nov 01 12:44:55.652072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.649100    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-4b54c1"
Nov 01 12:44:55.667422 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.660283    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8210b8fe-b7fc-4330-abbf-70771e446e13-kube-api-access-p8445" pod "8210b8fe-b7fc-4330-abbf-70771e446e13" (UID: "8210b8fe-b7fc-4330-abbf-70771e446e13"). InnerVolumeSpecName "kube-api-access-p8445". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:55.705655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.702405    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8445\" (UniqueName: \"kubernetes.io/projected/8210b8fe-b7fc-4330-abbf-70771e446e13-kube-api-access-p8445\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:55.740113 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:44:55.735720    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-4b54c1\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="8210b8fe-b7fc-4330-abbf-70771e446e13" pod="opendatahub/test-curl-7091-4b54c1"
Nov 01 12:44:55.755162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.755112    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:55.755499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.755449    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.812180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.805397    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.831537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.831489    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Nov 01 12:44:55.868219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.831228    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:44:55.868219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.858722    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:55.868219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.858772    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.868696 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.868653    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.885929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.885885    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:44:55.926279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:55.913427    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:56.083076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:56.075323    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:44:56.099612 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:56.099565    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:56.099910 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:56.099878    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:56.500520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:56.500476    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:56.505815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:56.501206    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:44:56.849354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:56.849257    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:56.949087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:56.868147    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:56.949087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:56.937385    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:44:56.970108 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:56.942954    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8210b8fe-b7fc-4330-abbf-70771e446e13" path="/var/lib/kubelet/pods/8210b8fe-b7fc-4330-abbf-70771e446e13/volumes"
Nov 01 12:44:56.970108 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:44:56.955242    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-4b54c1\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="8210b8fe-b7fc-4330-abbf-70771e446e13" pod="opendatahub/test-curl-7091-4b54c1"
Nov 01 12:44:57.098084 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.085562    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:57.360453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.360409    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x" event={"ID":"f9c32452-c11e-4a58-ac55-e9841a6775b3","Type":"ContainerDied","Data":"0ea4b997d944c1b40727d7dc33bd675de7b5989ca6d5c635a6e4def2cacfbb90"}
Nov 01 12:44:57.360744 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.360723    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ea4b997d944c1b40727d7dc33bd675de7b5989ca6d5c635a6e4def2cacfbb90"
Nov 01 12:44:57.360924 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.360897    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:57.423317 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.423279    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:57.496127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.477495    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-openshift-ca\") pod \"f9c32452-c11e-4a58-ac55-e9841a6775b3\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") "
Nov 01 12:44:57.507308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.507276    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c32452-c11e-4a58-ac55-e9841a6775b3-kube-api-access-xkqnx\" (UniqueName: \"kubernetes.io/projected/f9c32452-c11e-4a58-ac55-e9841a6775b3-kube-api-access-xkqnx\") pod \"f9c32452-c11e-4a58-ac55-e9841a6775b3\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") "
Nov 01 12:44:57.536453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.536412    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-xks-ca\") pod \"f9c32452-c11e-4a58-ac55-e9841a6775b3\" (UID: \"f9c32452-c11e-4a58-ac55-e9841a6775b3\") "
Nov 01 12:44:57.537828 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.480490    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-openshift-ca" pod "f9c32452-c11e-4a58-ac55-e9841a6775b3" (UID: "f9c32452-c11e-4a58-ac55-e9841a6775b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:44:57.537985 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.507154    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:57.603225 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.603185    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:57.656163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.643396    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:57.707078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.669226    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c32452-c11e-4a58-ac55-e9841a6775b3-kube-api-access-xkqnx" pod "f9c32452-c11e-4a58-ac55-e9841a6775b3" (UID: "f9c32452-c11e-4a58-ac55-e9841a6775b3"). InnerVolumeSpecName "kube-api-access-xkqnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:44:57.773073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.748309    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkqnx\" (UniqueName: \"kubernetes.io/projected/f9c32452-c11e-4a58-ac55-e9841a6775b3-kube-api-access-xkqnx\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:57.849244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.849201    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-xks-ca" pod "f9c32452-c11e-4a58-ac55-e9841a6775b3" (UID: "f9c32452-c11e-4a58-ac55-e9841a6775b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:44:57.850697 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.850668    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c32452-c11e-4a58-ac55-e9841a6775b3-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:44:57.953538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:57.953198    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:44:58.081076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:58.049264    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:58.710822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:58.257115    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:58.735539 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:58.494680    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:44:58.735539 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:58.562090    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:44:58.837463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:58.837414    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:59.028982 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.028932    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:59.707841 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.706267    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Nov 01 12:44:59.713293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.713248    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:44:59.713621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.713593    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:44:59.753800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.753769    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"
Nov 01 12:44:59.782370 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.782328    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:59.840578 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.838337    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:59.882155 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:44:59.882123    5725 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.169s"
Nov 01 12:44:59.882457 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.882441    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7091-8873ab"]
Nov 01 12:44:59.882621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.882611    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:44:59.882740 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.882729    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:44:59.885419 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.883586    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:44:59.901176 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.901144    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7091-8873ab" podUID="785396d2-da8c-411f-bcde-58deae9061c9" containerName="test-curl-7091-8873ab" containerID="cri-o://b2158fef763cb95dcc7d0fa83c4426f5e1ca4464866cb60a436e598122fe2614" gracePeriod=2
Nov 01 12:44:59.908489 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.908457    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7091-8873ab"]
Nov 01 12:44:59.908607 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.908516    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"]
Nov 01 12:44:59.908607 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.908559    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:44:59.910145 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.910121    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:44:59.910255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.910235    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:44:59.912508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.912486    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:44:59.914717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:44:59.914698    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c6ef091-x284p"]
Nov 01 12:45:00.064658 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.064611    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"]
Nov 01 12:45:00.067613 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:00.067572    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-8873ab\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="785396d2-da8c-411f-bcde-58deae9061c9" pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:45:00.071337 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.071311    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0dadb92-knkgt"]
Nov 01 12:45:00.195245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.195211    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"]
Nov 01 12:45:00.195886 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.195870    5725 cpu_manager.go:414] "containerMap: removing container" podUID="785396d2-da8c-411f-bcde-58deae9061c9" containerName="test-curl-7091-8873ab"
Nov 01 12:45:00.195937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.195887    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785396d2-da8c-411f-bcde-58deae9061c9" containerName="test-curl-7091-8873ab"
Nov 01 12:45:00.195937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.195913    5725 cpu_manager.go:414] "containerMap: removing container" podUID="6867e3c2-5bde-456d-a5d7-70640987a35f" containerName="revoke-keys"
Nov 01 12:45:00.195937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.195917    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6867e3c2-5bde-456d-a5d7-70640987a35f" containerName="revoke-keys"
Nov 01 12:45:00.196037 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.195947    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f9c32452-c11e-4a58-ac55-e9841a6775b3" containerName="revoke-keys"
Nov 01 12:45:00.196037 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.195952    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c32452-c11e-4a58-ac55-e9841a6775b3" containerName="revoke-keys"
Nov 01 12:45:00.196118 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.196065    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="785396d2-da8c-411f-bcde-58deae9061c9" containerName="test-curl-7091-8873ab"
Nov 01 12:45:00.196118 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.196078    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="6867e3c2-5bde-456d-a5d7-70640987a35f" containerName="revoke-keys"
Nov 01 12:45:00.196118 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.196089    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c32452-c11e-4a58-ac55-e9841a6775b3" containerName="revoke-keys"
Nov 01 12:45:00.200912 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.200891    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"]
Nov 01 12:45:00.201027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.201003    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:00.206171 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.206149    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"]
Nov 01 12:45:00.206312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.206258    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:00.211486 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.211467    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:00.261535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.261501    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"]
Nov 01 12:45:00.269879 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.269849    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:45:00.271656 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.271632    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:00.286173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.286143    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"]
Nov 01 12:45:00.300277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.292593    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:00.300277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.298227    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"]
Nov 01 12:45:00.315488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.315074    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd41da1-97ab-42d1-b59b-38b11003eb02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4\" (UID: \"7cd41da1-97ab-42d1-b59b-38b11003eb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:00.315488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.315237    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj\" (UID: \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:00.315488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.315315    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9mwp\" (UniqueName: \"kubernetes.io/projected/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-kube-api-access-k9mwp\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj\" (UID: \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:00.315488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.315340    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn\" (UID: \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:00.315488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.315363    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwjs\" (UniqueName: \"kubernetes.io/projected/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-kube-api-access-2qwjs\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn\" (UID: \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:00.315488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.315421    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqtdf\" (UniqueName: \"kubernetes.io/projected/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-kube-api-access-jqtdf\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p\" (UID: \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:00.315488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.315475    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p\" (UID: \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:00.316311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.315613    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtkcb\" (UniqueName: \"kubernetes.io/projected/7cd41da1-97ab-42d1-b59b-38b11003eb02-kube-api-access-qtkcb\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4\" (UID: \"7cd41da1-97ab-42d1-b59b-38b11003eb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:00.320899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.320412    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"]
Nov 01 12:45:00.327735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.327707    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"]
Nov 01 12:45:00.378778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.378736    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"]
Nov 01 12:45:00.390269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.387824    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"]
Nov 01 12:45:00.416755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.416717    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6fn\" (UniqueName: \"kubernetes.io/projected/b1b72167-4106-47d6-a77f-825927be629a-kube-api-access-xc6fn\") pod \"maas-api-key-cleanup-29892285-rs4mc\" (UID: \"b1b72167-4106-47d6-a77f-825927be629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:00.416921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.416774    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj\" (UID: \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:00.416921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.416901    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9mwp\" (UniqueName: \"kubernetes.io/projected/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-kube-api-access-k9mwp\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj\" (UID: \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:00.417126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.416940    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn\" (UID: \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:00.417126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.417071    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwjs\" (UniqueName: \"kubernetes.io/projected/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-kube-api-access-2qwjs\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn\" (UID: \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:00.417266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.417133    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqtdf\" (UniqueName: \"kubernetes.io/projected/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-kube-api-access-jqtdf\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p\" (UID: \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:00.417266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.417190    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p\" (UID: \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:00.417428 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.417278    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtkcb\" (UniqueName: \"kubernetes.io/projected/7cd41da1-97ab-42d1-b59b-38b11003eb02-kube-api-access-qtkcb\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4\" (UID: \"7cd41da1-97ab-42d1-b59b-38b11003eb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:00.417428 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.417310    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b72167-4106-47d6-a77f-825927be629a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892285-rs4mc\" (UID: \"b1b72167-4106-47d6-a77f-825927be629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:00.417555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.417484    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd41da1-97ab-42d1-b59b-38b11003eb02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4\" (UID: \"7cd41da1-97ab-42d1-b59b-38b11003eb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:00.417667 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.417644    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj\" (UID: \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:00.417758 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.417736    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn\" (UID: \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:00.418092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.418024    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p\" (UID: \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:00.418244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.418227    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd41da1-97ab-42d1-b59b-38b11003eb02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4\" (UID: \"7cd41da1-97ab-42d1-b59b-38b11003eb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:00.472536 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.472469    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtkcb\" (UniqueName: \"kubernetes.io/projected/7cd41da1-97ab-42d1-b59b-38b11003eb02-kube-api-access-qtkcb\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4\" (UID: \"7cd41da1-97ab-42d1-b59b-38b11003eb02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:00.472692 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.472621    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwjs\" (UniqueName: \"kubernetes.io/projected/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-kube-api-access-2qwjs\") pod \"maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn\" (UID: \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:00.472996 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.472975    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9mwp\" (UniqueName: \"kubernetes.io/projected/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-kube-api-access-k9mwp\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj\" (UID: \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:00.474116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.474028    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqtdf\" (UniqueName: \"kubernetes.io/projected/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-kube-api-access-jqtdf\") pod \"maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p\" (UID: \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:00.518380 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.518351    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:00.518935 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.518912    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6fn\" (UniqueName: \"kubernetes.io/projected/b1b72167-4106-47d6-a77f-825927be629a-kube-api-access-xc6fn\") pod \"maas-api-key-cleanup-29892285-rs4mc\" (UID: \"b1b72167-4106-47d6-a77f-825927be629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:00.519367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.519130    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b72167-4106-47d6-a77f-825927be629a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892285-rs4mc\" (UID: \"b1b72167-4106-47d6-a77f-825927be629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:00.519818 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.519797    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b72167-4106-47d6-a77f-825927be629a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892285-rs4mc\" (UID: \"b1b72167-4106-47d6-a77f-825927be629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:00.533112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.532835    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:00.565865 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.565784    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:00.576744 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.576685    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6fn\" (UniqueName: \"kubernetes.io/projected/b1b72167-4106-47d6-a77f-825927be629a-kube-api-access-xc6fn\") pod \"maas-api-key-cleanup-29892285-rs4mc\" (UID: \"b1b72167-4106-47d6-a77f-825927be629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:00.591164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.591129    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:00.656885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.639444    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:00.656885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.646641    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b902290-a726-4b0d-9993-b2c2b015dd57" path="/var/lib/kubelet/pods/9b902290-a726-4b0d-9993-b2c2b015dd57/volumes"
Nov 01 12:45:00.656885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.647435    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41e6112-9475-4ee9-9d6b-8437cebddc96" path="/var/lib/kubelet/pods/f41e6112-9475-4ee9-9d6b-8437cebddc96/volumes"
Nov 01 12:45:00.956608 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:00.956566    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"]
Nov 01 12:45:00.959908 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:45:00.959874    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd41da1_97ab_42d1_b59b_38b11003eb02.slice/crio-b5a35e26d1546369679ddc98e3735858d4b16e0228798cd19cdbc769d4245a86 WatchSource:0}: Error finding container b5a35e26d1546369679ddc98e3735858d4b16e0228798cd19cdbc769d4245a86: Status 404 returned error can't find the container with id b5a35e26d1546369679ddc98e3735858d4b16e0228798cd19cdbc769d4245a86
Nov 01 12:45:01.030593 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:01.030552    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:45:01.097948 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:01.097902    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-8873ab\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="785396d2-da8c-411f-bcde-58deae9061c9" pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:45:01.122592 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:01.122557    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"]
Nov 01 12:45:01.130884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:01.130853    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"]
Nov 01 12:45:01.135213 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:45:01.135126    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b72167_4106_47d6_a77f_825927be629a.slice/crio-52169b62f765dba4948f54c992ea4b163467fd22e5cd907450532e8397bffe2d WatchSource:0}: Error finding container 52169b62f765dba4948f54c992ea4b163467fd22e5cd907450532e8397bffe2d: Status 404 returned error can't find the container with id 52169b62f765dba4948f54c992ea4b163467fd22e5cd907450532e8397bffe2d
Nov 01 12:45:01.139133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:01.137620    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"]
Nov 01 12:45:01.219517 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:01.219432    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"]
Nov 01 12:45:01.408339 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:01.408282    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" event={"ID":"fae4f61a-067e-4ef6-b31c-2f17cbc9733b","Type":"ContainerStarted","Data":"b6f01eff8ad3ae794280c0350c8a99d000538b1cf3387edaa7ced18f82577aa8"}
Nov 01 12:45:01.410002 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:01.409974    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" event={"ID":"7cd41da1-97ab-42d1-b59b-38b11003eb02","Type":"ContainerStarted","Data":"b5a35e26d1546369679ddc98e3735858d4b16e0228798cd19cdbc769d4245a86"}
Nov 01 12:45:01.411493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:01.411453    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" event={"ID":"30883c72-aac0-4fd1-8fc4-6990f0f16e4b","Type":"ContainerStarted","Data":"a1bc621be26d11ca3d0a987c3ee15ea91bec551bc3b2aa78115d6e11fd5dd151"}
Nov 01 12:45:01.412945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:01.412923    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" event={"ID":"b1b72167-4106-47d6-a77f-825927be629a","Type":"ContainerStarted","Data":"52169b62f765dba4948f54c992ea4b163467fd22e5cd907450532e8397bffe2d"}
Nov 01 12:45:01.414431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:01.414408    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" event={"ID":"f555a9b4-8fed-48e6-baab-d4c3f28bb2df","Type":"ContainerStarted","Data":"ea577c544c3bdb324fe59c389ec71a191651b0cef1d8fd39eaaec32d2289f828"}
Nov 01 12:45:02.946020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.917753    5725 generic.go:356] "Generic (PLEG): container finished" podID="785396d2-da8c-411f-bcde-58deae9061c9" containerID="b2158fef763cb95dcc7d0fa83c4426f5e1ca4464866cb60a436e598122fe2614" exitCode=137
Nov 01 12:45:02.948633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.948593    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" event={"ID":"30883c72-aac0-4fd1-8fc4-6990f0f16e4b","Type":"ContainerStarted","Data":"5e689b227df95981cac475f0d69dc9f6f753017b14cc94b67a053b3976854dd4"}
Nov 01 12:45:02.949480 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.949456    5725 scope.go:122] "RemoveContainer" containerID="5e689b227df95981cac475f0d69dc9f6f753017b14cc94b67a053b3976854dd4"
Nov 01 12:45:02.952619 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.952591    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" event={"ID":"b1b72167-4106-47d6-a77f-825927be629a","Type":"ContainerStarted","Data":"e8fff55e8082e82608a63845c33259f5dc153de3db4eb7868d360c603bf01125"}
Nov 01 12:45:02.952969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.952949    5725 scope.go:122] "RemoveContainer" containerID="e8fff55e8082e82608a63845c33259f5dc153de3db4eb7868d360c603bf01125"
Nov 01 12:45:02.959668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.957449    5725 scope.go:122] "RemoveContainer" containerID="4c5833fe4b54f722dffd3e8f48fbfae4ad6ff9898089cdc965800ca3afb0cdb1"
Nov 01 12:45:02.959668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.957488    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" event={"ID":"f555a9b4-8fed-48e6-baab-d4c3f28bb2df","Type":"ContainerStarted","Data":"4c5833fe4b54f722dffd3e8f48fbfae4ad6ff9898089cdc965800ca3afb0cdb1"}
Nov 01 12:45:02.973961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.973824    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:45:02.974491 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.974456    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" event={"ID":"fae4f61a-067e-4ef6-b31c-2f17cbc9733b","Type":"ContainerStarted","Data":"21cec7e06362afbe43fc9fd63d54ea53d6ed6991222826121efeec21ec190fc8"}
Nov 01 12:45:02.975078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.975043    5725 scope.go:122] "RemoveContainer" containerID="21cec7e06362afbe43fc9fd63d54ea53d6ed6991222826121efeec21ec190fc8"
Nov 01 12:45:02.977773 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.977709    5725 generic.go:356] "Generic (PLEG): container finished" podID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerID="74a9814779114d01603779221b58ea81ef5447706bfc84d2760da756ad27229a" exitCode=60
Nov 01 12:45:02.977907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.977888    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" event={"ID":"7cd41da1-97ab-42d1-b59b-38b11003eb02","Type":"ContainerDied","Data":"74a9814779114d01603779221b58ea81ef5447706bfc84d2760da756ad27229a"}
Nov 01 12:45:02.978167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:02.978148    5725 scope.go:122] "RemoveContainer" containerID="74a9814779114d01603779221b58ea81ef5447706bfc84d2760da756ad27229a"
Nov 01 12:45:03.131075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.130589    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785396d2-da8c-411f-bcde-58deae9061c9-kube-api-access-qwncx\" (UniqueName: \"kubernetes.io/projected/785396d2-da8c-411f-bcde-58deae9061c9-kube-api-access-qwncx\") pod \"785396d2-da8c-411f-bcde-58deae9061c9\" (UID: \"785396d2-da8c-411f-bcde-58deae9061c9\") "
Nov 01 12:45:03.235581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.235537    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785396d2-da8c-411f-bcde-58deae9061c9-kube-api-access-qwncx" pod "785396d2-da8c-411f-bcde-58deae9061c9" (UID: "785396d2-da8c-411f-bcde-58deae9061c9"). InnerVolumeSpecName "kube-api-access-qwncx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:45:03.336094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.336033    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwncx\" (UniqueName: \"kubernetes.io/projected/785396d2-da8c-411f-bcde-58deae9061c9-kube-api-access-qwncx\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:03.989701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.989670    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:45:03.989701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.989689    5725 scope.go:122] "RemoveContainer" containerID="b2158fef763cb95dcc7d0fa83c4426f5e1ca4464866cb60a436e598122fe2614"
Nov 01 12:45:03.992354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.992322    5725 generic.go:356] "Generic (PLEG): container finished" podID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerID="5e689b227df95981cac475f0d69dc9f6f753017b14cc94b67a053b3976854dd4" exitCode=60
Nov 01 12:45:03.992502 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.992479    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" event={"ID":"30883c72-aac0-4fd1-8fc4-6990f0f16e4b","Type":"ContainerDied","Data":"5e689b227df95981cac475f0d69dc9f6f753017b14cc94b67a053b3976854dd4"}
Nov 01 12:45:03.992566 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.992538    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" event={"ID":"30883c72-aac0-4fd1-8fc4-6990f0f16e4b","Type":"ContainerStarted","Data":"15eacf1566db3fc8e2173c3b18b332c1069e8e0d6bf3475b8c1bf81166f53cc3"}
Nov 01 12:45:03.996026 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.995998    5725 generic.go:356] "Generic (PLEG): container finished" podID="b1b72167-4106-47d6-a77f-825927be629a" containerID="e8fff55e8082e82608a63845c33259f5dc153de3db4eb7868d360c603bf01125" exitCode=60
Nov 01 12:45:03.996182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.996103    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" event={"ID":"b1b72167-4106-47d6-a77f-825927be629a","Type":"ContainerDied","Data":"e8fff55e8082e82608a63845c33259f5dc153de3db4eb7868d360c603bf01125"}
Nov 01 12:45:03.996182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.996139    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" event={"ID":"b1b72167-4106-47d6-a77f-825927be629a","Type":"ContainerStarted","Data":"e5d8ca08ab316341693f6360faa95806be092700b947ed4c57eebee63fa2bda6"}
Nov 01 12:45:03.999235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.999207    5725 generic.go:356] "Generic (PLEG): container finished" podID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerID="4c5833fe4b54f722dffd3e8f48fbfae4ad6ff9898089cdc965800ca3afb0cdb1" exitCode=60
Nov 01 12:45:03.999375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.999296    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" event={"ID":"f555a9b4-8fed-48e6-baab-d4c3f28bb2df","Type":"ContainerDied","Data":"4c5833fe4b54f722dffd3e8f48fbfae4ad6ff9898089cdc965800ca3afb0cdb1"}
Nov 01 12:45:03.999375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:03.999345    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" event={"ID":"f555a9b4-8fed-48e6-baab-d4c3f28bb2df","Type":"ContainerStarted","Data":"0c924a8429fe1f47c0dbdd8114dc17addaf1710ec27ac6410cdf7281093a6ad1"}
Nov 01 12:45:04.001855 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.001827    5725 generic.go:356] "Generic (PLEG): container finished" podID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerID="21cec7e06362afbe43fc9fd63d54ea53d6ed6991222826121efeec21ec190fc8" exitCode=60
Nov 01 12:45:04.002036 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.002007    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" event={"ID":"fae4f61a-067e-4ef6-b31c-2f17cbc9733b","Type":"ContainerDied","Data":"21cec7e06362afbe43fc9fd63d54ea53d6ed6991222826121efeec21ec190fc8"}
Nov 01 12:45:04.002164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.002087    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" event={"ID":"fae4f61a-067e-4ef6-b31c-2f17cbc9733b","Type":"ContainerStarted","Data":"60ef5f08c73fbc285ee2ba9f53cc494463c45a817bc468ebce587dc4335bd268"}
Nov 01 12:45:04.004441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.004411    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" event={"ID":"7cd41da1-97ab-42d1-b59b-38b11003eb02","Type":"ContainerStarted","Data":"1d1e5cca8241a5ad7a5a3870b34a509a5bd4b511d43f6d26cb5d161a58c9f89d"}
Nov 01 12:45:04.105548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.105496    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" podStartSLOduration=4.105479899 podStartE2EDuration="4.105479899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:04.102233445 +0000 UTC m=+1345.609424661" watchObservedRunningTime="2026-11-01 12:45:04.105479899 +0000 UTC m=+1345.612671116"
Nov 01 12:45:04.226112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.226041    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" podStartSLOduration=4.226025385 podStartE2EDuration="4.226025385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:04.219399969 +0000 UTC m=+1345.726591185" watchObservedRunningTime="2026-11-01 12:45:04.226025385 +0000 UTC m=+1345.733216632"
Nov 01 12:45:04.347274 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.347222    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" podStartSLOduration=4.347204383 podStartE2EDuration="4.347204383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:04.339099769 +0000 UTC m=+1345.846290986" watchObservedRunningTime="2026-11-01 12:45:04.347204383 +0000 UTC m=+1345.854395599"
Nov 01 12:45:04.460741 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.460679    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" podStartSLOduration=4.460661148 podStartE2EDuration="4.460661148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:04.459807928 +0000 UTC m=+1345.966999158" watchObservedRunningTime="2026-11-01 12:45:04.460661148 +0000 UTC m=+1345.967852366"
Nov 01 12:45:04.515832 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:04.515783    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-8873ab\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="785396d2-da8c-411f-bcde-58deae9061c9" pod="opendatahub/test-curl-7091-8873ab"
Nov 01 12:45:04.618179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.618100    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785396d2-da8c-411f-bcde-58deae9061c9" path="/var/lib/kubelet/pods/785396d2-da8c-411f-bcde-58deae9061c9/volumes"
Nov 01 12:45:04.636632 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:04.636574    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" podStartSLOduration=4.636556476 podStartE2EDuration="4.636556476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:04.633569977 +0000 UTC m=+1346.140761195" watchObservedRunningTime="2026-11-01 12:45:04.636556476 +0000 UTC m=+1346.143747744"
Nov 01 12:45:05.028189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.028154    5725 generic.go:356] "Generic (PLEG): container finished" podID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerID="1d1e5cca8241a5ad7a5a3870b34a509a5bd4b511d43f6d26cb5d161a58c9f89d" exitCode=60
Nov 01 12:45:05.028680 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.028242    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" event={"ID":"7cd41da1-97ab-42d1-b59b-38b11003eb02","Type":"ContainerDied","Data":"1d1e5cca8241a5ad7a5a3870b34a509a5bd4b511d43f6d26cb5d161a58c9f89d"}
Nov 01 12:45:05.028680 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.028356    5725 scope.go:122] "RemoveContainer" containerID="74a9814779114d01603779221b58ea81ef5447706bfc84d2760da756ad27229a"
Nov 01 12:45:05.028762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.028742    5725 scope.go:122] "RemoveContainer" containerID="1d1e5cca8241a5ad7a5a3870b34a509a5bd4b511d43f6d26cb5d161a58c9f89d"
Nov 01 12:45:05.029208 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:05.029186    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4_odh-ai-gateway-infra(7cd41da1-97ab-42d1-b59b-38b11003eb02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02"
Nov 01 12:45:05.032805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.032766    5725 generic.go:356] "Generic (PLEG): container finished" podID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerID="15eacf1566db3fc8e2173c3b18b332c1069e8e0d6bf3475b8c1bf81166f53cc3" exitCode=60
Nov 01 12:45:05.032939 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.032871    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" event={"ID":"30883c72-aac0-4fd1-8fc4-6990f0f16e4b","Type":"ContainerDied","Data":"15eacf1566db3fc8e2173c3b18b332c1069e8e0d6bf3475b8c1bf81166f53cc3"}
Nov 01 12:45:05.033428 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.033408    5725 scope.go:122] "RemoveContainer" containerID="15eacf1566db3fc8e2173c3b18b332c1069e8e0d6bf3475b8c1bf81166f53cc3"
Nov 01 12:45:05.033859 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:05.033831    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj_odh-ai-gateway-infra(30883c72-aac0-4fd1-8fc4-6990f0f16e4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b"
Nov 01 12:45:05.035159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.035122    5725 generic.go:356] "Generic (PLEG): container finished" podID="b1b72167-4106-47d6-a77f-825927be629a" containerID="e5d8ca08ab316341693f6360faa95806be092700b947ed4c57eebee63fa2bda6" exitCode=60
Nov 01 12:45:05.035267 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.035198    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" event={"ID":"b1b72167-4106-47d6-a77f-825927be629a","Type":"ContainerDied","Data":"e5d8ca08ab316341693f6360faa95806be092700b947ed4c57eebee63fa2bda6"}
Nov 01 12:45:05.035476 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.035459    5725 scope.go:122] "RemoveContainer" containerID="e5d8ca08ab316341693f6360faa95806be092700b947ed4c57eebee63fa2bda6"
Nov 01 12:45:05.035858 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:05.035830    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892285-rs4mc_odh-ai-gateway-infra(b1b72167-4106-47d6-a77f-825927be629a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" podUID="b1b72167-4106-47d6-a77f-825927be629a"
Nov 01 12:45:05.037532 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.037508    5725 generic.go:356] "Generic (PLEG): container finished" podID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerID="0c924a8429fe1f47c0dbdd8114dc17addaf1710ec27ac6410cdf7281093a6ad1" exitCode=60
Nov 01 12:45:05.037855 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.037836    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" event={"ID":"f555a9b4-8fed-48e6-baab-d4c3f28bb2df","Type":"ContainerDied","Data":"0c924a8429fe1f47c0dbdd8114dc17addaf1710ec27ac6410cdf7281093a6ad1"}
Nov 01 12:45:05.038035 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.038019    5725 scope.go:122] "RemoveContainer" containerID="0c924a8429fe1f47c0dbdd8114dc17addaf1710ec27ac6410cdf7281093a6ad1"
Nov 01 12:45:05.038479 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:05.038454    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn_odh-ai-gateway-infra(f555a9b4-8fed-48e6-baab-d4c3f28bb2df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df"
Nov 01 12:45:05.040063 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.040030    5725 generic.go:356] "Generic (PLEG): container finished" podID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerID="60ef5f08c73fbc285ee2ba9f53cc494463c45a817bc468ebce587dc4335bd268" exitCode=60
Nov 01 12:45:05.040151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.040089    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" event={"ID":"fae4f61a-067e-4ef6-b31c-2f17cbc9733b","Type":"ContainerDied","Data":"60ef5f08c73fbc285ee2ba9f53cc494463c45a817bc468ebce587dc4335bd268"}
Nov 01 12:45:05.040383 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.040366    5725 scope.go:122] "RemoveContainer" containerID="60ef5f08c73fbc285ee2ba9f53cc494463c45a817bc468ebce587dc4335bd268"
Nov 01 12:45:05.040687 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:05.040661    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p_odh-ai-gateway-infra(fae4f61a-067e-4ef6-b31c-2f17cbc9733b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b"
Nov 01 12:45:05.047037 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.047010    5725 scope.go:122] "RemoveContainer" containerID="5e689b227df95981cac475f0d69dc9f6f753017b14cc94b67a053b3976854dd4"
Nov 01 12:45:05.063062 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.063023    5725 scope.go:122] "RemoveContainer" containerID="e8fff55e8082e82608a63845c33259f5dc153de3db4eb7868d360c603bf01125"
Nov 01 12:45:05.080992 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.080960    5725 scope.go:122] "RemoveContainer" containerID="4c5833fe4b54f722dffd3e8f48fbfae4ad6ff9898089cdc965800ca3afb0cdb1"
Nov 01 12:45:05.096878 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:05.096858    5725 scope.go:122] "RemoveContainer" containerID="21cec7e06362afbe43fc9fd63d54ea53d6ed6991222826121efeec21ec190fc8"
Nov 01 12:45:06.054433 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:06.054396    5725 scope.go:122] "RemoveContainer" containerID="60ef5f08c73fbc285ee2ba9f53cc494463c45a817bc468ebce587dc4335bd268"
Nov 01 12:45:06.054963 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:06.054838    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p_odh-ai-gateway-infra(fae4f61a-067e-4ef6-b31c-2f17cbc9733b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b"
Nov 01 12:45:06.056922 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:06.056894    5725 scope.go:122] "RemoveContainer" containerID="1d1e5cca8241a5ad7a5a3870b34a509a5bd4b511d43f6d26cb5d161a58c9f89d"
Nov 01 12:45:06.057379 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:06.057340    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4_odh-ai-gateway-infra(7cd41da1-97ab-42d1-b59b-38b11003eb02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02"
Nov 01 12:45:06.059636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:06.059613    5725 scope.go:122] "RemoveContainer" containerID="15eacf1566db3fc8e2173c3b18b332c1069e8e0d6bf3475b8c1bf81166f53cc3"
Nov 01 12:45:06.060035 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:06.060009    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj_odh-ai-gateway-infra(30883c72-aac0-4fd1-8fc4-6990f0f16e4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b"
Nov 01 12:45:06.062376 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:06.062347    5725 scope.go:122] "RemoveContainer" containerID="e5d8ca08ab316341693f6360faa95806be092700b947ed4c57eebee63fa2bda6"
Nov 01 12:45:06.066158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:06.066132    5725 scope.go:122] "RemoveContainer" containerID="0c924a8429fe1f47c0dbdd8114dc17addaf1710ec27ac6410cdf7281093a6ad1"
Nov 01 12:45:06.066531 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:06.066505    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn_odh-ai-gateway-infra(f555a9b4-8fed-48e6-baab-d4c3f28bb2df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df"
Nov 01 12:45:06.068201 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:06.068167    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892285-rs4mc_odh-ai-gateway-infra(b1b72167-4106-47d6-a77f-825927be629a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" podUID="b1b72167-4106-47d6-a77f-825927be629a"
Nov 01 12:45:09.143981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.143939    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"]
Nov 01 12:45:09.151539 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.151514    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.277002 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.273890    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b2ccbb55-openshift-default-dockercfg-4vnjb\""
Nov 01 12:45:09.319074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.315993    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-data\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.319074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.316138    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-certs\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.319074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.316171    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istiod-ca-cert\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.319074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.316344    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-socket\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.319074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.316381    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-credential-socket\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.319074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.316432    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-podinfo\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.319074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.316497    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-envoy\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.319074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.316529    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-token\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.319074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.316580    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhr89\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-kube-api-access-bhr89\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.339445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.339347    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"]
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.426126    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-socket\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.426185    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-credential-socket\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.426233    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-podinfo\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.426274    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-envoy\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.426309    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-token\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.426361    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhr89\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-kube-api-access-bhr89\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.426418    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-data\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.426559    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-certs\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.426595    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istiod-ca-cert\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.427576    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istiod-ca-cert\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.428079    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-socket\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.440119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.428334    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-credential-socket\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.466959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.460960    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-data\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.466959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.461655    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-certs\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.466959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.463949    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-podinfo\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.479136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.473225    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-envoy\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.517073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.516507    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-token\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.524142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.521679    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhr89\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-kube-api-access-bhr89\") pod \"e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") " pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:09.524142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:09.522068    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:10.077153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:10.077114    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"]
Nov 01 12:45:10.080966 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:45:10.080939    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ef04e5_718b_43dc_9544_b0b4be25f9a6.slice/crio-8645dcb6649024c07b025dd34af34bea7b2afef6172ab51af4ba2e7f5c0d97d9 WatchSource:0}: Error finding container 8645dcb6649024c07b025dd34af34bea7b2afef6172ab51af4ba2e7f5c0d97d9: Status 404 returned error can't find the container with id 8645dcb6649024c07b025dd34af34bea7b2afef6172ab51af4ba2e7f5c0d97d9
Nov 01 12:45:10.084457 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:10.084413    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:45:10.084556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:10.084496    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:45:10.084556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:10.084528    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:45:10.114003 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:10.113960    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" event={"ID":"69ef04e5-718b-43dc-9544-b0b4be25f9a6","Type":"ContainerStarted","Data":"8645dcb6649024c07b025dd34af34bea7b2afef6172ab51af4ba2e7f5c0d97d9"}
Nov 01 12:45:11.126698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:11.126660    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" event={"ID":"69ef04e5-718b-43dc-9544-b0b4be25f9a6","Type":"ContainerStarted","Data":"4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2"}
Nov 01 12:45:11.522863 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:11.522829    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:11.530228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:11.530203    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:11.729706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:11.729645    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" podStartSLOduration=2.7295968200000003 podStartE2EDuration="2.72959682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:11.341365915 +0000 UTC m=+1352.848557133" watchObservedRunningTime="2026-11-01 12:45:11.72959682 +0000 UTC m=+1353.236788038"
Nov 01 12:45:12.139217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:12.139182    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:12.140688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:12.140658    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:45:15.609618 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:15.609587    5725 scope.go:122] "RemoveContainer" containerID="0c924a8429fe1f47c0dbdd8114dc17addaf1710ec27ac6410cdf7281093a6ad1"
Nov 01 12:45:15.610043 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:15.609736    5725 scope.go:122] "RemoveContainer" containerID="1d1e5cca8241a5ad7a5a3870b34a509a5bd4b511d43f6d26cb5d161a58c9f89d"
Nov 01 12:45:15.610043 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:15.609855    5725 scope.go:122] "RemoveContainer" containerID="e5d8ca08ab316341693f6360faa95806be092700b947ed4c57eebee63fa2bda6"
Nov 01 12:45:16.071241 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:16.071208    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"]
Nov 01 12:45:16.118617 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:16.118583    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee81217e-ljnsg"]
Nov 01 12:45:16.181006 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:16.180871    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" event={"ID":"7cd41da1-97ab-42d1-b59b-38b11003eb02","Type":"ContainerStarted","Data":"0a56461cb009504d3d54525190608b64efad2bc897f1eb1aa7787fa8115955a6"}
Nov 01 12:45:16.184275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:16.184250    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" event={"ID":"b1b72167-4106-47d6-a77f-825927be629a","Type":"ContainerStarted","Data":"111f47c6c94f0370eeeeb59ff9d224e651a79d366c132a172bbe2fc2146024ab"}
Nov 01 12:45:16.187279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:16.187246    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" event={"ID":"f555a9b4-8fed-48e6-baab-d4c3f28bb2df","Type":"ContainerStarted","Data":"fa0ef4d9db52964f86e484e6dd1c599eb1c7142bcd72bd6cf33720ab04d414b4"}
Nov 01 12:45:16.618297 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:16.617642    5725 scope.go:122] "RemoveContainer" containerID="60ef5f08c73fbc285ee2ba9f53cc494463c45a817bc468ebce587dc4335bd268"
Nov 01 12:45:16.619885 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:16.619339    5725 scope.go:122] "RemoveContainer" containerID="15eacf1566db3fc8e2173c3b18b332c1069e8e0d6bf3475b8c1bf81166f53cc3"
Nov 01 12:45:16.652526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:16.652476    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768d1620-f31c-4161-bb42-c8f3d27c24a6" path="/var/lib/kubelet/pods/768d1620-f31c-4161-bb42-c8f3d27c24a6/volumes"
Nov 01 12:45:17.055993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.051152    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"]
Nov 01 12:45:17.127819 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.127733    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.128137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.127963    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"]
Nov 01 12:45:17.145036 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.145007    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.145234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.145088    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b738d395-2a03-407d-8846-459e221674ba-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.145234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.145119    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.145234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.145158    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9qp\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-kube-api-access-sl9qp\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.145234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.145188    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.145461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.145248    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.145461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.145317    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.145461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.145375    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b738d395-2a03-407d-8846-459e221674ba-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.145461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.145444    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.204203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.204105    5725 generic.go:356] "Generic (PLEG): container finished" podID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerID="0a56461cb009504d3d54525190608b64efad2bc897f1eb1aa7787fa8115955a6" exitCode=60
Nov 01 12:45:17.204203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.204158    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" event={"ID":"7cd41da1-97ab-42d1-b59b-38b11003eb02","Type":"ContainerDied","Data":"0a56461cb009504d3d54525190608b64efad2bc897f1eb1aa7787fa8115955a6"}
Nov 01 12:45:17.204465 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.204237    5725 scope.go:122] "RemoveContainer" containerID="1d1e5cca8241a5ad7a5a3870b34a509a5bd4b511d43f6d26cb5d161a58c9f89d"
Nov 01 12:45:17.204828 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.204808    5725 scope.go:122] "RemoveContainer" containerID="0a56461cb009504d3d54525190608b64efad2bc897f1eb1aa7787fa8115955a6"
Nov 01 12:45:17.207770 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:17.206794    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4_odh-ai-gateway-infra(7cd41da1-97ab-42d1-b59b-38b11003eb02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02"
Nov 01 12:45:17.209650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.209625    5725 generic.go:356] "Generic (PLEG): container finished" podID="b1b72167-4106-47d6-a77f-825927be629a" containerID="111f47c6c94f0370eeeeb59ff9d224e651a79d366c132a172bbe2fc2146024ab" exitCode=60
Nov 01 12:45:17.209788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.209710    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" event={"ID":"b1b72167-4106-47d6-a77f-825927be629a","Type":"ContainerDied","Data":"111f47c6c94f0370eeeeb59ff9d224e651a79d366c132a172bbe2fc2146024ab"}
Nov 01 12:45:17.210427 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.210377    5725 scope.go:122] "RemoveContainer" containerID="111f47c6c94f0370eeeeb59ff9d224e651a79d366c132a172bbe2fc2146024ab"
Nov 01 12:45:17.210927 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:17.210898    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892285-rs4mc_odh-ai-gateway-infra(b1b72167-4106-47d6-a77f-825927be629a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" podUID="b1b72167-4106-47d6-a77f-825927be629a"
Nov 01 12:45:17.216258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.215313    5725 generic.go:356] "Generic (PLEG): container finished" podID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerID="fa0ef4d9db52964f86e484e6dd1c599eb1c7142bcd72bd6cf33720ab04d414b4" exitCode=60
Nov 01 12:45:17.216258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.215361    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" event={"ID":"f555a9b4-8fed-48e6-baab-d4c3f28bb2df","Type":"ContainerDied","Data":"fa0ef4d9db52964f86e484e6dd1c599eb1c7142bcd72bd6cf33720ab04d414b4"}
Nov 01 12:45:17.216258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.215801    5725 scope.go:122] "RemoveContainer" containerID="fa0ef4d9db52964f86e484e6dd1c599eb1c7142bcd72bd6cf33720ab04d414b4"
Nov 01 12:45:17.216258 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:17.216201    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn_odh-ai-gateway-infra(f555a9b4-8fed-48e6-baab-d4c3f28bb2df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df"
Nov 01 12:45:17.237340 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.237219    5725 scope.go:122] "RemoveContainer" containerID="e5d8ca08ab316341693f6360faa95806be092700b947ed4c57eebee63fa2bda6"
Nov 01 12:45:17.261303 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.261275    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.261539 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.261523    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b738d395-2a03-407d-8846-459e221674ba-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.261691 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.261676    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.261838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.261819    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9qp\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-kube-api-access-sl9qp\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.261957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.261939    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.263353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.263327    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.263617 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.263594    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.263813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.263794    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b738d395-2a03-407d-8846-459e221674ba-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.264079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.264045    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.264764 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.264722    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.266568 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.266478    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.267990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.267962    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.268592 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.268461    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b738d395-2a03-407d-8846-459e221674ba-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.268883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.268857    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.269901 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.269882    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.277080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.277036    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b738d395-2a03-407d-8846-459e221674ba-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.318236 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.318213    5725 scope.go:122] "RemoveContainer" containerID="0c924a8429fe1f47c0dbdd8114dc17addaf1710ec27ac6410cdf7281093a6ad1"
Nov 01 12:45:17.332409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.332365    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9qp\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-kube-api-access-sl9qp\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.332605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.332416    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.341307 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.341281    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"]
Nov 01 12:45:17.449004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.448961    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:17.451312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.451272    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"]
Nov 01 12:45:17.552538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.552509    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"]
Nov 01 12:45:17.903677 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.903646    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"]
Nov 01 12:45:17.955931 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:45:17.955886    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb738d395_2a03_407d_8846_459e221674ba.slice/crio-ce1d63aff79870f49dd17882a962cdf78aabe786374219921897335fa5b3bcd5 WatchSource:0}: Error finding container ce1d63aff79870f49dd17882a962cdf78aabe786374219921897335fa5b3bcd5: Status 404 returned error can't find the container with id ce1d63aff79870f49dd17882a962cdf78aabe786374219921897335fa5b3bcd5
Nov 01 12:45:17.958817 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.958785    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:45:17.958899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.958855    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:45:17.958899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:17.958887    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:45:18.230279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.230228    5725 generic.go:356] "Generic (PLEG): container finished" podID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerID="14dd80f87b7761aea94ac125f21823c568f50821861a6f5c9b32fd79b03fea81" exitCode=60
Nov 01 12:45:18.230450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.230297    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" event={"ID":"30883c72-aac0-4fd1-8fc4-6990f0f16e4b","Type":"ContainerDied","Data":"14dd80f87b7761aea94ac125f21823c568f50821861a6f5c9b32fd79b03fea81"}
Nov 01 12:45:18.230450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.230391    5725 scope.go:122] "RemoveContainer" containerID="15eacf1566db3fc8e2173c3b18b332c1069e8e0d6bf3475b8c1bf81166f53cc3"
Nov 01 12:45:18.230786 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.230769    5725 scope.go:122] "RemoveContainer" containerID="14dd80f87b7761aea94ac125f21823c568f50821861a6f5c9b32fd79b03fea81"
Nov 01 12:45:18.231243 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:18.231081    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj_odh-ai-gateway-infra(30883c72-aac0-4fd1-8fc4-6990f0f16e4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b"
Nov 01 12:45:18.237109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.237045    5725 generic.go:356] "Generic (PLEG): container finished" podID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerID="0b5ea15d4d7a105e1165ee5e8ae62f0c4dc2c2ceebf7861380c9d7e861145207" exitCode=60
Nov 01 12:45:18.237219 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.237098    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" event={"ID":"fae4f61a-067e-4ef6-b31c-2f17cbc9733b","Type":"ContainerDied","Data":"0b5ea15d4d7a105e1165ee5e8ae62f0c4dc2c2ceebf7861380c9d7e861145207"}
Nov 01 12:45:18.237849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.237830    5725 scope.go:122] "RemoveContainer" containerID="0b5ea15d4d7a105e1165ee5e8ae62f0c4dc2c2ceebf7861380c9d7e861145207"
Nov 01 12:45:18.238339 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:18.238315    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p_odh-ai-gateway-infra(fae4f61a-067e-4ef6-b31c-2f17cbc9733b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b"
Nov 01 12:45:18.239401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.239324    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" event={"ID":"b738d395-2a03-407d-8846-459e221674ba","Type":"ContainerStarted","Data":"e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848"}
Nov 01 12:45:18.239401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.239372    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" event={"ID":"b738d395-2a03-407d-8846-459e221674ba","Type":"ContainerStarted","Data":"ce1d63aff79870f49dd17882a962cdf78aabe786374219921897335fa5b3bcd5"}
Nov 01 12:45:18.249220 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.248915    5725 scope.go:122] "RemoveContainer" containerID="60ef5f08c73fbc285ee2ba9f53cc494463c45a817bc468ebce587dc4335bd268"
Nov 01 12:45:18.456489 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.456433    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:18.517544 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.516741    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:18.570933 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.570905    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:18.589418 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.589384    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-maas-api-service-ca\") pod \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\" (UID: \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\") "
Nov 01 12:45:18.589661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.589587    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-kube-api-access-2qwjs\" (UniqueName: \"kubernetes.io/projected/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-kube-api-access-2qwjs\") pod \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\" (UID: \"f555a9b4-8fed-48e6-baab-d4c3f28bb2df\") "
Nov 01 12:45:18.589661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.589632    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b72167-4106-47d6-a77f-825927be629a-kube-api-access-xc6fn\" (UniqueName: \"kubernetes.io/projected/b1b72167-4106-47d6-a77f-825927be629a-kube-api-access-xc6fn\") pod \"b1b72167-4106-47d6-a77f-825927be629a\" (UID: \"b1b72167-4106-47d6-a77f-825927be629a\") "
Nov 01 12:45:18.590490 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.590169    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b72167-4106-47d6-a77f-825927be629a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b72167-4106-47d6-a77f-825927be629a-maas-api-service-ca\") pod \"b1b72167-4106-47d6-a77f-825927be629a\" (UID: \"b1b72167-4106-47d6-a77f-825927be629a\") "
Nov 01 12:45:18.590991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.590962    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b72167-4106-47d6-a77f-825927be629a-maas-api-service-ca" pod "b1b72167-4106-47d6-a77f-825927be629a" (UID: "b1b72167-4106-47d6-a77f-825927be629a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:45:18.591398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.591374    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-maas-api-service-ca" pod "f555a9b4-8fed-48e6-baab-d4c3f28bb2df" (UID: "f555a9b4-8fed-48e6-baab-d4c3f28bb2df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:45:18.594265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.594229    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b72167-4106-47d6-a77f-825927be629a-kube-api-access-xc6fn" pod "b1b72167-4106-47d6-a77f-825927be629a" (UID: "b1b72167-4106-47d6-a77f-825927be629a"). InnerVolumeSpecName "kube-api-access-xc6fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:45:18.594463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.594436    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-kube-api-access-2qwjs" pod "f555a9b4-8fed-48e6-baab-d4c3f28bb2df" (UID: "f555a9b4-8fed-48e6-baab-d4c3f28bb2df"). InnerVolumeSpecName "kube-api-access-2qwjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:45:18.675579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.675509    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:18.697090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.693564    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd41da1-97ab-42d1-b59b-38b11003eb02-kube-api-access-qtkcb\" (UniqueName: \"kubernetes.io/projected/7cd41da1-97ab-42d1-b59b-38b11003eb02-kube-api-access-qtkcb\") pod \"7cd41da1-97ab-42d1-b59b-38b11003eb02\" (UID: \"7cd41da1-97ab-42d1-b59b-38b11003eb02\") "
Nov 01 12:45:18.697090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.693839    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd41da1-97ab-42d1-b59b-38b11003eb02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd41da1-97ab-42d1-b59b-38b11003eb02-maas-api-service-ca\") pod \"7cd41da1-97ab-42d1-b59b-38b11003eb02\" (UID: \"7cd41da1-97ab-42d1-b59b-38b11003eb02\") "
Nov 01 12:45:18.697090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.694680    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b72167-4106-47d6-a77f-825927be629a-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:18.697090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.694708    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:18.697090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.694723    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qwjs\" (UniqueName: \"kubernetes.io/projected/f555a9b4-8fed-48e6-baab-d4c3f28bb2df-kube-api-access-2qwjs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:18.697090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.694738    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc6fn\" (UniqueName: \"kubernetes.io/projected/b1b72167-4106-47d6-a77f-825927be629a-kube-api-access-xc6fn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:18.701958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.700492    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd41da1-97ab-42d1-b59b-38b11003eb02-kube-api-access-qtkcb" pod "7cd41da1-97ab-42d1-b59b-38b11003eb02" (UID: "7cd41da1-97ab-42d1-b59b-38b11003eb02"). InnerVolumeSpecName "kube-api-access-qtkcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:45:18.701958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.701647    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd41da1-97ab-42d1-b59b-38b11003eb02-maas-api-service-ca" pod "7cd41da1-97ab-42d1-b59b-38b11003eb02" (UID: "7cd41da1-97ab-42d1-b59b-38b11003eb02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:45:18.796152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.796064    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtkcb\" (UniqueName: \"kubernetes.io/projected/7cd41da1-97ab-42d1-b59b-38b11003eb02-kube-api-access-qtkcb\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:18.796152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.796092    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cd41da1-97ab-42d1-b59b-38b11003eb02-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:18.972705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:18.972650    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" podStartSLOduration=2.972636494 podStartE2EDuration="2.972636494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:18.968937863 +0000 UTC m=+1360.476129081" watchObservedRunningTime="2026-11-01 12:45:18.972636494 +0000 UTC m=+1360.479827707"
Nov 01 12:45:19.268544 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.268515    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"
Nov 01 12:45:19.269781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.268434    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc" event={"ID":"b1b72167-4106-47d6-a77f-825927be629a","Type":"ContainerDied","Data":"52169b62f765dba4948f54c992ea4b163467fd22e5cd907450532e8397bffe2d"}
Nov 01 12:45:19.269991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.269973    5725 scope.go:122] "RemoveContainer" containerID="111f47c6c94f0370eeeeb59ff9d224e651a79d366c132a172bbe2fc2146024ab"
Nov 01 12:45:19.270431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.270399    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn" event={"ID":"f555a9b4-8fed-48e6-baab-d4c3f28bb2df","Type":"ContainerDied","Data":"ea577c544c3bdb324fe59c389ec71a191651b0cef1d8fd39eaaec32d2289f828"}
Nov 01 12:45:19.270546 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.270495    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"
Nov 01 12:45:19.275796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.275079    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"
Nov 01 12:45:19.276036 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.275998    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4" event={"ID":"7cd41da1-97ab-42d1-b59b-38b11003eb02","Type":"ContainerDied","Data":"b5a35e26d1546369679ddc98e3735858d4b16e0228798cd19cdbc769d4245a86"}
Nov 01 12:45:19.289204 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.289183    5725 scope.go:122] "RemoveContainer" containerID="fa0ef4d9db52964f86e484e6dd1c599eb1c7142bcd72bd6cf33720ab04d414b4"
Nov 01 12:45:19.302154 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.302127    5725 scope.go:122] "RemoveContainer" containerID="0a56461cb009504d3d54525190608b64efad2bc897f1eb1aa7787fa8115955a6"
Nov 01 12:45:19.369034 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.368993    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"]
Nov 01 12:45:19.458117 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.457800    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 01 12:45:19.458117 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.457852    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 01 12:45:19.525869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.525730    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"]
Nov 01 12:45:19.564196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.564144    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892285-hpqb4"]
Nov 01 12:45:19.626716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.626692    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:19.713472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.713440    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-kube-api-access-k9mwp\" (UniqueName: \"kubernetes.io/projected/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-kube-api-access-k9mwp\") pod \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\" (UID: \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\") "
Nov 01 12:45:19.713654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.713637    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-maas-api-service-ca\") pod \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\" (UID: \"30883c72-aac0-4fd1-8fc4-6990f0f16e4b\") "
Nov 01 12:45:19.714230 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.714201    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-maas-api-service-ca" pod "30883c72-aac0-4fd1-8fc4-6990f0f16e4b" (UID: "30883c72-aac0-4fd1-8fc4-6990f0f16e4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:45:19.714358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.714336    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:19.716303 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.716275    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-kube-api-access-k9mwp" pod "30883c72-aac0-4fd1-8fc4-6990f0f16e4b" (UID: "30883c72-aac0-4fd1-8fc4-6990f0f16e4b"). InnerVolumeSpecName "kube-api-access-k9mwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:45:19.793258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.793164    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"]
Nov 01 12:45:19.798327 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.798302    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-94c94b-29892285-g58pn"]
Nov 01 12:45:19.815661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.815629    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9mwp\" (UniqueName: \"kubernetes.io/projected/30883c72-aac0-4fd1-8fc4-6990f0f16e4b-kube-api-access-k9mwp\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:19.884071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.880675    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"]
Nov 01 12:45:19.918157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.918116    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"]
Nov 01 12:45:19.923500 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:19.923468    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892285-rs4mc"]
Nov 01 12:45:20.177073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.173549    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:20.220145 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.220103    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-kube-api-access-jqtdf\" (UniqueName: \"kubernetes.io/projected/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-kube-api-access-jqtdf\") pod \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\" (UID: \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\") "
Nov 01 12:45:20.220345 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.220318    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-maas-api-service-ca\") pod \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\" (UID: \"fae4f61a-067e-4ef6-b31c-2f17cbc9733b\") "
Nov 01 12:45:20.220735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.220710    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-maas-api-service-ca" pod "fae4f61a-067e-4ef6-b31c-2f17cbc9733b" (UID: "fae4f61a-067e-4ef6-b31c-2f17cbc9733b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:45:20.221097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.221076    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:20.222747 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.222722    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-kube-api-access-jqtdf" pod "fae4f61a-067e-4ef6-b31c-2f17cbc9733b" (UID: "fae4f61a-067e-4ef6-b31c-2f17cbc9733b"). InnerVolumeSpecName "kube-api-access-jqtdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:45:20.296746 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.296719    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"
Nov 01 12:45:20.296897 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.296710    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p" event={"ID":"fae4f61a-067e-4ef6-b31c-2f17cbc9733b","Type":"ContainerDied","Data":"b6f01eff8ad3ae794280c0350c8a99d000538b1cf3387edaa7ced18f82577aa8"}
Nov 01 12:45:20.296897 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.296824    5725 scope.go:122] "RemoveContainer" containerID="0b5ea15d4d7a105e1165ee5e8ae62f0c4dc2c2ceebf7861380c9d7e861145207"
Nov 01 12:45:20.299818 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.299787    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj" event={"ID":"30883c72-aac0-4fd1-8fc4-6990f0f16e4b","Type":"ContainerDied","Data":"a1bc621be26d11ca3d0a987c3ee15ea91bec551bc3b2aa78115d6e11fd5dd151"}
Nov 01 12:45:20.299945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.299823    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"
Nov 01 12:45:20.319689 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.319615    5725 scope.go:122] "RemoveContainer" containerID="14dd80f87b7761aea94ac125f21823c568f50821861a6f5c9b32fd79b03fea81"
Nov 01 12:45:20.322475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.322446    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqtdf\" (UniqueName: \"kubernetes.io/projected/fae4f61a-067e-4ef6-b31c-2f17cbc9733b-kube-api-access-jqtdf\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:20.463281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.463245    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 01 12:45:20.463859 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.463747    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 01 12:45:20.464713 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.464689    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"]
Nov 01 12:45:20.469904 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.469880    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9514b0-29892285-lnb5p"]
Nov 01 12:45:20.575177 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.575143    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"]
Nov 01 12:45:20.583380 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.583350    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7427d-29892285-mndwj"]
Nov 01 12:45:20.617851 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.617820    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" path="/var/lib/kubelet/pods/30883c72-aac0-4fd1-8fc4-6990f0f16e4b/volumes"
Nov 01 12:45:20.618342 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.618328    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" path="/var/lib/kubelet/pods/7cd41da1-97ab-42d1-b59b-38b11003eb02/volumes"
Nov 01 12:45:20.618736 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.618723    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b72167-4106-47d6-a77f-825927be629a" path="/var/lib/kubelet/pods/b1b72167-4106-47d6-a77f-825927be629a/volumes"
Nov 01 12:45:20.619441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.619419    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" path="/var/lib/kubelet/pods/f555a9b4-8fed-48e6-baab-d4c3f28bb2df/volumes"
Nov 01 12:45:20.619869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:20.619851    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" path="/var/lib/kubelet/pods/fae4f61a-067e-4ef6-b31c-2f17cbc9733b/volumes"
Nov 01 12:45:21.450074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:21.449403    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 01 12:45:21.450074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:21.449461    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 01 12:45:22.155106 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.155018    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"]
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156094    5725 cpu_manager.go:414] "containerMap: removing container" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156117    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156135    5725 cpu_manager.go:414] "containerMap: removing container" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156144    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156161    5725 cpu_manager.go:414] "containerMap: removing container" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156169    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156183    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b1b72167-4106-47d6-a77f-825927be629a" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156190    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b72167-4106-47d6-a77f-825927be629a" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156215    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156222    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156244    5725 cpu_manager.go:414] "containerMap: removing container" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156252    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156265    5725 cpu_manager.go:414] "containerMap: removing container" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156272    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156296    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b1b72167-4106-47d6-a77f-825927be629a" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156305    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b72167-4106-47d6-a77f-825927be629a" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156338    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156345    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156356    5725 cpu_manager.go:414] "containerMap: removing container" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156363    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156541    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156552    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156562    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156571    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156580    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156591    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b72167-4106-47d6-a77f-825927be629a" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156599    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b72167-4106-47d6-a77f-825927be629a" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156611    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156620    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156631    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156640    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156651    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156661    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b72167-4106-47d6-a77f-825927be629a" containerName="cleanup"
Nov 01 12:45:22.191163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.156671    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerName="cleanup"
Nov 01 12:45:22.484597 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.484560    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 01 12:45:22.485247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.485208    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 01 12:45:22.557179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.557146    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"]
Nov 01 12:45:22.557546 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.557525    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:22.631712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.631257    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvp5\" (UniqueName: \"kubernetes.io/projected/44747563-edf3-414b-8e28-e027b06dd779-kube-api-access-bmvp5\") pod \"maas-api-revoke-sub-c24e8e9b-624t2\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:22.631712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.631342    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44747563-edf3-414b-8e28-e027b06dd779-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c24e8e9b-624t2\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:22.631712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.631637    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44747563-edf3-414b-8e28-e027b06dd779-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c24e8e9b-624t2\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:22.651012 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.650980    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:45:22.699335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.699277    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:45:22.733400 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.733362    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvp5\" (UniqueName: \"kubernetes.io/projected/44747563-edf3-414b-8e28-e027b06dd779-kube-api-access-bmvp5\") pod \"maas-api-revoke-sub-c24e8e9b-624t2\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:22.733693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.733674    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44747563-edf3-414b-8e28-e027b06dd779-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c24e8e9b-624t2\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:22.734273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.734250    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44747563-edf3-414b-8e28-e027b06dd779-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c24e8e9b-624t2\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:22.742900 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.739667    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44747563-edf3-414b-8e28-e027b06dd779-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c24e8e9b-624t2\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:22.743473 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.743451    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44747563-edf3-414b-8e28-e027b06dd779-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c24e8e9b-624t2\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:22.849335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:22.849299    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvp5\" (UniqueName: \"kubernetes.io/projected/44747563-edf3-414b-8e28-e027b06dd779-kube-api-access-bmvp5\") pod \"maas-api-revoke-sub-c24e8e9b-624t2\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:23.041786 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:23.041422    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:23.462093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:23.461596    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 01 12:45:23.462093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:23.461654    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 01 12:45:23.474734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:23.474654    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"]
Nov 01 12:45:23.474734 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:45:23.474683    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44747563_edf3_414b_8e28_e027b06dd779.slice/crio-a312af7724f24b5440779654b80a557b2e02291d142294d366ab32ffa644c26f WatchSource:0}: Error finding container a312af7724f24b5440779654b80a557b2e02291d142294d366ab32ffa644c26f: Status 404 returned error can't find the container with id a312af7724f24b5440779654b80a557b2e02291d142294d366ab32ffa644c26f
Nov 01 12:45:23.589463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:23.589396    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2" event={"ID":"44747563-edf3-414b-8e28-e027b06dd779","Type":"ContainerStarted","Data":"a312af7724f24b5440779654b80a557b2e02291d142294d366ab32ffa644c26f"}
Nov 01 12:45:24.449792 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:24.449746    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 01 12:45:24.450074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:24.449816    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 01 12:45:24.604331 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:24.604304    5725 generic.go:356] "Generic (PLEG): container finished" podID="44747563-edf3-414b-8e28-e027b06dd779" containerID="d344b3418ebd8bb74f352488f04391ee12a579cf386d0e9515308f7b901fce2a" exitCode=0
Nov 01 12:45:24.604715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:24.604386    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2" event={"ID":"44747563-edf3-414b-8e28-e027b06dd779","Type":"ContainerDied","Data":"d344b3418ebd8bb74f352488f04391ee12a579cf386d0e9515308f7b901fce2a"}
Nov 01 12:45:26.052862 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.052815    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:26.153781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.153741    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44747563-edf3-414b-8e28-e027b06dd779-kube-api-access-bmvp5\" (UniqueName: \"kubernetes.io/projected/44747563-edf3-414b-8e28-e027b06dd779-kube-api-access-bmvp5\") pod \"44747563-edf3-414b-8e28-e027b06dd779\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") "
Nov 01 12:45:26.153958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.153867    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44747563-edf3-414b-8e28-e027b06dd779-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44747563-edf3-414b-8e28-e027b06dd779-maas-api-openshift-ca\") pod \"44747563-edf3-414b-8e28-e027b06dd779\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") "
Nov 01 12:45:26.154122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.154106    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44747563-edf3-414b-8e28-e027b06dd779-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44747563-edf3-414b-8e28-e027b06dd779-maas-api-xks-ca\") pod \"44747563-edf3-414b-8e28-e027b06dd779\" (UID: \"44747563-edf3-414b-8e28-e027b06dd779\") "
Nov 01 12:45:26.154369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.154340    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44747563-edf3-414b-8e28-e027b06dd779-maas-api-openshift-ca" pod "44747563-edf3-414b-8e28-e027b06dd779" (UID: "44747563-edf3-414b-8e28-e027b06dd779"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:45:26.154724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.154704    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44747563-edf3-414b-8e28-e027b06dd779-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:26.156585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.156463    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44747563-edf3-414b-8e28-e027b06dd779-kube-api-access-bmvp5" pod "44747563-edf3-414b-8e28-e027b06dd779" (UID: "44747563-edf3-414b-8e28-e027b06dd779"). InnerVolumeSpecName "kube-api-access-bmvp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:45:26.156585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.156550    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44747563-edf3-414b-8e28-e027b06dd779-maas-api-xks-ca" pod "44747563-edf3-414b-8e28-e027b06dd779" (UID: "44747563-edf3-414b-8e28-e027b06dd779"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:45:26.223117 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.223074    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:26.223544 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.223517    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:26.224674 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.224652    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:45:26.256721 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.256630    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44747563-edf3-414b-8e28-e027b06dd779-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:26.256987 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.256933    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmvp5\" (UniqueName: \"kubernetes.io/projected/44747563-edf3-414b-8e28-e027b06dd779-kube-api-access-bmvp5\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:26.658286 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.658255    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"
Nov 01 12:45:26.658286 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.658270    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2" event={"ID":"44747563-edf3-414b-8e28-e027b06dd779","Type":"ContainerDied","Data":"a312af7724f24b5440779654b80a557b2e02291d142294d366ab32ffa644c26f"}
Nov 01 12:45:26.658514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:26.658360    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a312af7724f24b5440779654b80a557b2e02291d142294d366ab32ffa644c26f"
Nov 01 12:45:31.431792 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:31.431705    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:45:31.489223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:31.488179    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.582189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:31.581147    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.704260 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:31.704149    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.709595 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:31.709554    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.888200 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:31.888152    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.919075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:31.910143    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:45:31.967370 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:31.967260    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:45:32.258075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:32.251143    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:32.682128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:32.673465    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:45:32.682128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:32.675453    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:45:32.682128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:32.675868    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:45:32.682128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:32.675915    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:45:32.707395 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:32.705405    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Nov 01 12:45:32.929929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:32.929885    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 12:45:32.962382 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:32.936139    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:45:33.005434 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.003088    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:45:33.034598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.017485    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:45:33.034598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.017553    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:45:33.034598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.017596    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:45:33.034598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.017632    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:45:33.690085 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.684122    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:45:33.690085 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.687120    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:45:33.838862 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.838821    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"]
Nov 01 12:45:33.859076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.855196    5725 cpu_manager.go:414] "containerMap: removing container" podUID="44747563-edf3-414b-8e28-e027b06dd779" containerName="revoke-keys"
Nov 01 12:45:33.879126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.879089    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44747563-edf3-414b-8e28-e027b06dd779" containerName="revoke-keys"
Nov 01 12:45:33.879409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.879390    5725 cpu_manager.go:414] "containerMap: removing container" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerName="cleanup"
Nov 01 12:45:33.879516 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.879503    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30883c72-aac0-4fd1-8fc4-6990f0f16e4b" containerName="cleanup"
Nov 01 12:45:33.879616 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.879605    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b1b72167-4106-47d6-a77f-825927be629a" containerName="cleanup"
Nov 01 12:45:33.885534 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.885502    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b72167-4106-47d6-a77f-825927be629a" containerName="cleanup"
Nov 01 12:45:33.885728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.885713    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerName="cleanup"
Nov 01 12:45:33.889110 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.889082    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerName="cleanup"
Nov 01 12:45:33.889300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.889286    5725 cpu_manager.go:414] "containerMap: removing container" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerName="cleanup"
Nov 01 12:45:33.893210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.893185    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae4f61a-067e-4ef6-b31c-2f17cbc9733b" containerName="cleanup"
Nov 01 12:45:33.893358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.893342    5725 cpu_manager.go:414] "containerMap: removing container" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerName="cleanup"
Nov 01 12:45:33.893443 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.893431    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd41da1-97ab-42d1-b59b-38b11003eb02" containerName="cleanup"
Nov 01 12:45:33.916095 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.893753    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f555a9b4-8fed-48e6-baab-d4c3f28bb2df" containerName="cleanup"
Nov 01 12:45:33.916095 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:33.903124    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="44747563-edf3-414b-8e28-e027b06dd779" containerName="revoke-keys"
Nov 01 12:45:34.008326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.007129    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:45:34.034416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.013140    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:45:34.034416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.014119    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Nov 01 12:45:34.036360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.036319    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:34.081686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.081653    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"]
Nov 01 12:45:34.100206 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.082124    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:34.187029 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.186998    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:45:34.188306 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.188247    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:45:34.268293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.268190    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:34.497077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.449694    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c0430f-8n87d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:34.497077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.449960    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c0430f-8n87d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:34.497077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.462280    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cdkx\" (UniqueName: \"kubernetes.io/projected/f32afab9-a145-48f6-8559-d612b675f08d-kube-api-access-4cdkx\") pod \"maas-api-revoke-sub-22c0430f-8n87d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:34.668131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.617995    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cdkx\" (UniqueName: \"kubernetes.io/projected/f32afab9-a145-48f6-8559-d612b675f08d-kube-api-access-4cdkx\") pod \"maas-api-revoke-sub-22c0430f-8n87d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:34.668131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.618765    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c0430f-8n87d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:34.668131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.618996    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c0430f-8n87d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:34.668131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.652360    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c0430f-8n87d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:34.825094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.761149    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 12:45:34.825094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.763313    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:34.975480 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:34.975433    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c0430f-8n87d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:35.069188 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.069129    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:35.082460 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.082411    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:45:35.168208 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.083060    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:45:35.173571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.083285    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Nov 01 12:45:35.378129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.362458    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cdkx\" (UniqueName: \"kubernetes.io/projected/f32afab9-a145-48f6-8559-d612b675f08d-kube-api-access-4cdkx\") pod \"maas-api-revoke-sub-22c0430f-8n87d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:35.709490 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.675217    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:35.715310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.715261    5725 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:35.755394 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.755345    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:35.769088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.761608    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:35.819442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:35.819397    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:45:36.096287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:36.096228    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.283076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:36.261452    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 01 12:45:36.283076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:36.279684    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.291398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:36.291358    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Nov 01 12:45:36.973601 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:36.973552    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.974938 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:36.974892    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": context deadline exceeded"
Nov 01 12:45:37.055071 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:45:37.045022    5725 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.281s"
Nov 01 12:45:37.221727 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.221685    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:37.307343 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.307301    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:37.385074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.382452    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"]
Nov 01 12:45:37.401588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.401563    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 12:45:37.401799 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.401789    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 12:45:37.401900 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.401891    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 12:45:37.402849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.402828    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.421988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.418401    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"]
Nov 01 12:45:37.428466 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.426289    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 12:45:37.430891 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.429965    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 12:45:37.431999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.431317    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 12:45:37.449657 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.449628    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c057128e-776c-4691-9904-597c45635c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ed9b47-s9vz6\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.452073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.452035    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c057128e-776c-4691-9904-597c45635c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ed9b47-s9vz6\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.452392 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.452375    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjrh\" (UniqueName: \"kubernetes.io/projected/c057128e-776c-4691-9904-597c45635c81-kube-api-access-jxjrh\") pod \"maas-api-revoke-sub-42ed9b47-s9vz6\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.563092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.553482    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c057128e-776c-4691-9904-597c45635c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ed9b47-s9vz6\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.563092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.554131    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c057128e-776c-4691-9904-597c45635c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ed9b47-s9vz6\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.563092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.554137    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c057128e-776c-4691-9904-597c45635c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ed9b47-s9vz6\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.563092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.554297    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjrh\" (UniqueName: \"kubernetes.io/projected/c057128e-776c-4691-9904-597c45635c81-kube-api-access-jxjrh\") pod \"maas-api-revoke-sub-42ed9b47-s9vz6\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.569687 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.569571    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c057128e-776c-4691-9904-597c45635c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ed9b47-s9vz6\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.624071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.621487    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjrh\" (UniqueName: \"kubernetes.io/projected/c057128e-776c-4691-9904-597c45635c81-kube-api-access-jxjrh\") pod \"maas-api-revoke-sub-42ed9b47-s9vz6\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.749218 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.749134    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:37.891061 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:37.891021    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"]
Nov 01 12:45:38.187226 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:38.187191    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"]
Nov 01 12:45:38.195277 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:45:38.195210    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc057128e_776c_4691_9904_597c45635c81.slice/crio-071d01a0805398e78c6dbcbb46d546550aab6591efeaa4cf8434bb5e6e83145e WatchSource:0}: Error finding container 071d01a0805398e78c6dbcbb46d546550aab6591efeaa4cf8434bb5e6e83145e: Status 404 returned error can't find the container with id 071d01a0805398e78c6dbcbb46d546550aab6591efeaa4cf8434bb5e6e83145e
Nov 01 12:45:38.416484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:38.416380    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6" event={"ID":"c057128e-776c-4691-9904-597c45635c81","Type":"ContainerStarted","Data":"071d01a0805398e78c6dbcbb46d546550aab6591efeaa4cf8434bb5e6e83145e"}
Nov 01 12:45:38.418424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:38.418390    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d" event={"ID":"f32afab9-a145-48f6-8559-d612b675f08d","Type":"ContainerStarted","Data":"52ffa171bedc02bc30ddfeff4fc622d48740b01616336501bf4a2a11df3b90fb"}
Nov 01 12:45:38.418572 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:38.418450    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d" event={"ID":"f32afab9-a145-48f6-8559-d612b675f08d","Type":"ContainerStarted","Data":"da505ea7d0b22280e40ad6f4a5f56e29d0b1789c48ed3ce42c2eb43f4deba29f"}
Nov 01 12:45:38.539379 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:38.539316    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d" podStartSLOduration=5.539300142 podStartE2EDuration="5.539300142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:38.538582338 +0000 UTC m=+1380.045773561" watchObservedRunningTime="2026-11-01 12:45:38.539300142 +0000 UTC m=+1380.046491353"
Nov 01 12:45:39.033668 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.033626    5725 scope.go:122] "RemoveContainer" containerID="2b6b08508a06d8dd86709bcb83389dd5931be2c13b60d9337366325956adad21"
Nov 01 12:45:39.045650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.045624    5725 scope.go:122] "RemoveContainer" containerID="c3543a8a58c7ee209d8a02a1965a4e0a645f3a04ba8dc130035eaf569ae5a69e"
Nov 01 12:45:39.056629 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.056608    5725 scope.go:122] "RemoveContainer" containerID="0e1988ce89a9e2e4e67a6e8c7aa1559ffee207762cc328e358634664aa207aba"
Nov 01 12:45:39.067473 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.067449    5725 scope.go:122] "RemoveContainer" containerID="26422fc1b9d1eb108bd858643e21b586e35b132df8c29b62ba5f2dc554eb4096"
Nov 01 12:45:39.080886 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.080857    5725 scope.go:122] "RemoveContainer" containerID="8431628f2549606e94150bec33f4f0bd0f5197bfa4c866b2317f1186ef028251"
Nov 01 12:45:39.093475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.093441    5725 scope.go:122] "RemoveContainer" containerID="0db09bfc911669882e58a83fe92c5b33d6c828e1544b25f6d7bd4d59ccd9dbad"
Nov 01 12:45:39.104818 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.104794    5725 scope.go:122] "RemoveContainer" containerID="97a5de8fa283c4c42414d7fcb39544d1cc84a74c71ded0b437f6331968616062"
Nov 01 12:45:39.115617 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.115591    5725 scope.go:122] "RemoveContainer" containerID="ef8c458f8ea9635136e4712b23adbdd5f85374a764fa2f51b06c16bc816aa68d"
Nov 01 12:45:39.132040 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.132018    5725 scope.go:122] "RemoveContainer" containerID="84ea4363d7b64ac36c38773bbbec36ed66869ff3bfedb3104f2d381c69abe8e1"
Nov 01 12:45:39.142652 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.142633    5725 scope.go:122] "RemoveContainer" containerID="b0a296e6c2e761a495c13f07b78cc6d9f6af521314e273b11efc823c3fd3952a"
Nov 01 12:45:39.152840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.152812    5725 scope.go:122] "RemoveContainer" containerID="c9039916d341d8c0d48a5292a3cac2fb91a5bc11ffb5dbc301eee0509b7f6c29"
Nov 01 12:45:39.431869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.431774    5725 generic.go:356] "Generic (PLEG): container finished" podID="c057128e-776c-4691-9904-597c45635c81" containerID="4d499e3d8eaa4d814098b6f53e45b8ff8be4f6e5796b57db5ebfc0958380d445" exitCode=0
Nov 01 12:45:39.431869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.431857    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6" event={"ID":"c057128e-776c-4691-9904-597c45635c81","Type":"ContainerDied","Data":"4d499e3d8eaa4d814098b6f53e45b8ff8be4f6e5796b57db5ebfc0958380d445"}
Nov 01 12:45:39.434014 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.433982    5725 generic.go:356] "Generic (PLEG): container finished" podID="f32afab9-a145-48f6-8559-d612b675f08d" containerID="52ffa171bedc02bc30ddfeff4fc622d48740b01616336501bf4a2a11df3b90fb" exitCode=0
Nov 01 12:45:39.434194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:39.434077    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d" event={"ID":"f32afab9-a145-48f6-8559-d612b675f08d","Type":"ContainerDied","Data":"52ffa171bedc02bc30ddfeff4fc622d48740b01616336501bf4a2a11df3b90fb"}
Nov 01 12:45:40.711216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.711128    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:40.748109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.748044    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:40.748480 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.748456    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-openshift-ca\") pod \"f32afab9-a145-48f6-8559-d612b675f08d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") "
Nov 01 12:45:40.748572 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.748555    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-xks-ca\") pod \"f32afab9-a145-48f6-8559-d612b675f08d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") "
Nov 01 12:45:40.748785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.748767    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32afab9-a145-48f6-8559-d612b675f08d-kube-api-access-4cdkx\" (UniqueName: \"kubernetes.io/projected/f32afab9-a145-48f6-8559-d612b675f08d-kube-api-access-4cdkx\") pod \"f32afab9-a145-48f6-8559-d612b675f08d\" (UID: \"f32afab9-a145-48f6-8559-d612b675f08d\") "
Nov 01 12:45:40.749212 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.749185    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-openshift-ca" pod "f32afab9-a145-48f6-8559-d612b675f08d" (UID: "f32afab9-a145-48f6-8559-d612b675f08d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:45:40.749547 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.749525    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:40.753506 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.753479    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-xks-ca" pod "f32afab9-a145-48f6-8559-d612b675f08d" (UID: "f32afab9-a145-48f6-8559-d612b675f08d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:45:40.753715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.753681    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32afab9-a145-48f6-8559-d612b675f08d-kube-api-access-4cdkx" pod "f32afab9-a145-48f6-8559-d612b675f08d" (UID: "f32afab9-a145-48f6-8559-d612b675f08d"). InnerVolumeSpecName "kube-api-access-4cdkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:45:40.850906 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.850864    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c057128e-776c-4691-9904-597c45635c81-kube-api-access-jxjrh\" (UniqueName: \"kubernetes.io/projected/c057128e-776c-4691-9904-597c45635c81-kube-api-access-jxjrh\") pod \"c057128e-776c-4691-9904-597c45635c81\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") "
Nov 01 12:45:40.851131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.850952    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c057128e-776c-4691-9904-597c45635c81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c057128e-776c-4691-9904-597c45635c81-maas-api-xks-ca\") pod \"c057128e-776c-4691-9904-597c45635c81\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") "
Nov 01 12:45:40.851131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.851084    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c057128e-776c-4691-9904-597c45635c81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c057128e-776c-4691-9904-597c45635c81-maas-api-openshift-ca\") pod \"c057128e-776c-4691-9904-597c45635c81\" (UID: \"c057128e-776c-4691-9904-597c45635c81\") "
Nov 01 12:45:40.851713 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.851686    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cdkx\" (UniqueName: \"kubernetes.io/projected/f32afab9-a145-48f6-8559-d612b675f08d-kube-api-access-4cdkx\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:40.851713 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.851714    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32afab9-a145-48f6-8559-d612b675f08d-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:40.852113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.852087    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c057128e-776c-4691-9904-597c45635c81-maas-api-openshift-ca" pod "c057128e-776c-4691-9904-597c45635c81" (UID: "c057128e-776c-4691-9904-597c45635c81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:45:40.858624 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.858589    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c057128e-776c-4691-9904-597c45635c81-kube-api-access-jxjrh" pod "c057128e-776c-4691-9904-597c45635c81" (UID: "c057128e-776c-4691-9904-597c45635c81"). InnerVolumeSpecName "kube-api-access-jxjrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:45:40.860567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.860537    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c057128e-776c-4691-9904-597c45635c81-maas-api-xks-ca" pod "c057128e-776c-4691-9904-597c45635c81" (UID: "c057128e-776c-4691-9904-597c45635c81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:45:40.953463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.953419    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxjrh\" (UniqueName: \"kubernetes.io/projected/c057128e-776c-4691-9904-597c45635c81-kube-api-access-jxjrh\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:40.953463 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.953464    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c057128e-776c-4691-9904-597c45635c81-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:40.953712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:40.953481    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c057128e-776c-4691-9904-597c45635c81-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:45:41.461780 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:41.461731    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6" event={"ID":"c057128e-776c-4691-9904-597c45635c81","Type":"ContainerDied","Data":"071d01a0805398e78c6dbcbb46d546550aab6591efeaa4cf8434bb5e6e83145e"}
Nov 01 12:45:41.461990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:41.461753    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"
Nov 01 12:45:41.461990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:41.461846    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="071d01a0805398e78c6dbcbb46d546550aab6591efeaa4cf8434bb5e6e83145e"
Nov 01 12:45:41.463869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:41.463819    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d" event={"ID":"f32afab9-a145-48f6-8559-d612b675f08d","Type":"ContainerDied","Data":"da505ea7d0b22280e40ad6f4a5f56e29d0b1789c48ed3ce42c2eb43f4deba29f"}
Nov 01 12:45:41.464022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:41.463885    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da505ea7d0b22280e40ad6f4a5f56e29d0b1789c48ed3ce42c2eb43f4deba29f"
Nov 01 12:45:41.464022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:41.463947    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"
Nov 01 12:45:47.159968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.159935    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"]
Nov 01 12:45:47.160639 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.160622    5725 cpu_manager.go:414] "containerMap: removing container" podUID="c057128e-776c-4691-9904-597c45635c81" containerName="revoke-keys"
Nov 01 12:45:47.160688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.160640    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c057128e-776c-4691-9904-597c45635c81" containerName="revoke-keys"
Nov 01 12:45:47.160688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.160683    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f32afab9-a145-48f6-8559-d612b675f08d" containerName="revoke-keys"
Nov 01 12:45:47.160766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.160689    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32afab9-a145-48f6-8559-d612b675f08d" containerName="revoke-keys"
Nov 01 12:45:47.160800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.160795    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="c057128e-776c-4691-9904-597c45635c81" containerName="revoke-keys"
Nov 01 12:45:47.160832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.160810    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32afab9-a145-48f6-8559-d612b675f08d" containerName="revoke-keys"
Nov 01 12:45:47.164794 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.164767    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.263082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.263009    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.263287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.263121    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.263287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.263240    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d86828-3b34-49c4-82d9-8d90a1908e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.263287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.263275    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.263448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.263316    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.264235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.263450    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.264235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.263483    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrxl\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-kube-api-access-5lrxl\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.264235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.263567    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.264235 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.263622    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.285748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.285709    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"]
Nov 01 12:45:47.364949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.364904    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.364949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.364946    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrxl\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-kube-api-access-5lrxl\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365007    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365096    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365186    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365271    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365327    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d86828-3b34-49c4-82d9-8d90a1908e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365374    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365541 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365414    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365636    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365620    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.365883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365800    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.366111 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.366092    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d86828-3b34-49c4-82d9-8d90a1908e65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.366185 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.365959    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.367937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.367915    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.368190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.368172    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.421162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.421075    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.421338 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.421202    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrxl\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-kube-api-access-5lrxl\") pod \"maas-default-gateway-openshift-default-7df854bf67-b8568\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.482119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.482081    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:47.933078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.933029    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"]
Nov 01 12:45:47.936688 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:45:47.936660    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d86828_3b34_49c4_82d9_8d90a1908e65.slice/crio-0f44cf0986caa1cf153c080dee96418b8dac5fa92dfc520b0e96d6601eada3fd WatchSource:0}: Error finding container 0f44cf0986caa1cf153c080dee96418b8dac5fa92dfc520b0e96d6601eada3fd: Status 404 returned error can't find the container with id 0f44cf0986caa1cf153c080dee96418b8dac5fa92dfc520b0e96d6601eada3fd
Nov 01 12:45:47.939331 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.939294    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:45:47.939428 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.939364    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:45:47.939428 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:47.939394    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:45:48.572654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:48.572604    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" event={"ID":"71d86828-3b34-49c4-82d9-8d90a1908e65","Type":"ContainerStarted","Data":"d11645f3d35f1ef8a2f2fa79969d3d63646ca91f1edccab0bf2b94fe1309f33e"}
Nov 01 12:45:48.573145 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:48.572668    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" event={"ID":"71d86828-3b34-49c4-82d9-8d90a1908e65","Type":"ContainerStarted","Data":"0f44cf0986caa1cf153c080dee96418b8dac5fa92dfc520b0e96d6601eada3fd"}
Nov 01 12:45:48.791715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:48.791667    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" podStartSLOduration=1.791648183 podStartE2EDuration="1.791648183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:48.74472903 +0000 UTC m=+1390.251920239" watchObservedRunningTime="2026-11-01 12:45:48.791648183 +0000 UTC m=+1390.298839397"
Nov 01 12:45:49.482591 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:49.482552    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:49.484689 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:49.484661    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b8568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 12:45:49.484826 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:49.484712    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 12:45:50.482805 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:50.482764    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b8568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 12:45:50.483237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:50.482830    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 12:45:51.482749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:51.482712    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b8568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 12:45:51.482968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:51.482777    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 12:45:52.483491 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:52.483453    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b8568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 12:45:52.483907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:52.483522    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 12:45:53.482619 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:53.482585    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b8568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 12:45:53.482808 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:53.482645    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 12:45:54.482809 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:54.482759    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b8568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 12:45:54.483249 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:54.482843    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 12:45:55.482793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:55.482756    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b8568 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 12:45:55.482972 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:55.482810    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 12:45:56.913089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:56.913039    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:56.914244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:56.914217    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:45:56.915059 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:45:56.915030    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:46:01.038130 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:46:01.038096    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:46:02.333077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.323470    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"]
Nov 01 12:46:02.373317 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.373283    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"]
Nov 01 12:46:02.375178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.373617    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.383121 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.382323    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.476459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.476425    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtz5f\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-kube-api-access-mtz5f\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.514339 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.514305    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3faca-0aac-4627-9b21-759db7314974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.546414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.546373    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.547456 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.547182    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.547950 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.547929    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.548276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.548258    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.548667 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.548649    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.558388 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.558358    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.565674 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.565219    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.709389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.701843    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb3faca-0aac-4627-9b21-759db7314974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.709389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.701936    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.709389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.701982    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwvs\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-kube-api-access-vtwvs\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.709389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.702018    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.709389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.702042    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.709389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.702108    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.709389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.702183    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.709389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.702221    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.709389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.702253    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.892567    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtz5f\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-kube-api-access-mtz5f\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.892621    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3faca-0aac-4627-9b21-759db7314974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.892681    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.892708    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.892761    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.892789    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.892809    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.892898    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.892920    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893002    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb3faca-0aac-4627-9b21-759db7314974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893079    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893125    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwvs\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-kube-api-access-vtwvs\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.893800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893171    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.893800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893194    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.893800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893239    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893295    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.893800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893323    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893499    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.893800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.893691    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.894273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.894017    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:02.917077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.894515    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3faca-0aac-4627-9b21-759db7314974-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:02.946077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.928285    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"]
Nov 01 12:46:02.979142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.977223    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtz5f\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-kube-api-access-mtz5f\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:03.002168 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:02.986294    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.010484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.003924    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.032456 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.011127    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.032799 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.011787    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.032935 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.028090    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.033034 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.028973    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:03.033178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.029263    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:03.033282 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.032720    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 12:46:03.070840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.070696    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.070840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.070765    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"]
Nov 01 12:46:03.093753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.093716    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.120074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.116273    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:03.132025 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.131951    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwvs\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-kube-api-access-vtwvs\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.159183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.152544    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-nbf9w\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.179217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.177730    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb3faca-0aac-4627-9b21-759db7314974-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:03.179217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.177829    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:46:03.179217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.178484    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:03.180348 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.180323    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-sscmt\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:03.277178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.277143    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:03.438087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.430852    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:46:03.501708 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:03.494570    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:46:04.349448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:04.344958    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:46:04.349448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:04.345085    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:46:04.349448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:04.345130    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:46:04.454182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:04.448411    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"]
Nov 01 12:46:04.502073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:04.501921    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"]
Nov 01 12:46:04.511750 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:46:04.511705    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb3faca_0aac_4627_9b21_759db7314974.slice/crio-e7049ead99fce29052c97ec5a5c48d840d466863c7352ab31efc05768599a9be WatchSource:0}: Error finding container e7049ead99fce29052c97ec5a5c48d840d466863c7352ab31efc05768599a9be: Status 404 returned error can't find the container with id e7049ead99fce29052c97ec5a5c48d840d466863c7352ab31efc05768599a9be
Nov 01 12:46:04.517776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:04.516616    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:46:04.517776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:04.516706    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:46:04.517776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:04.516751    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:46:05.394005 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:05.393953    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" event={"ID":"0bb3faca-0aac-4627-9b21-759db7314974","Type":"ContainerStarted","Data":"1e56258900e2a58adcc35be42accb200f24c050de049cc5d1c8a439394d86ab4"}
Nov 01 12:46:05.402106 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:05.401875    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" event={"ID":"0bb3faca-0aac-4627-9b21-759db7314974","Type":"ContainerStarted","Data":"e7049ead99fce29052c97ec5a5c48d840d466863c7352ab31efc05768599a9be"}
Nov 01 12:46:05.436969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:05.436918    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" event={"ID":"dd50dc72-ea77-4b81-9be5-db8ff0c0978e","Type":"ContainerStarted","Data":"01cc0820b4fdfcfd507fcfcdbc9c78313d306321b9b02aff98d9d9ea3f47b026"}
Nov 01 12:46:05.437184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:05.437005    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" event={"ID":"dd50dc72-ea77-4b81-9be5-db8ff0c0978e","Type":"ContainerStarted","Data":"2e9332abc113bdc516d2382ff6979e8955b25a418ee9c0f34dcc95729a3ebc90"}
Nov 01 12:46:05.632425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:05.632363    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podStartSLOduration=3.632344005 podStartE2EDuration="3.632344005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:46:05.629264702 +0000 UTC m=+1407.136455954" watchObservedRunningTime="2026-11-01 12:46:05.632344005 +0000 UTC m=+1407.139535222"
Nov 01 12:46:05.794075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:05.780742    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podStartSLOduration=3.780716131 podStartE2EDuration="3.780716131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:46:05.762617033 +0000 UTC m=+1407.269808254" watchObservedRunningTime="2026-11-01 12:46:05.780716131 +0000 UTC m=+1407.287907352"
Nov 01 12:46:06.145266 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:06.145173    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:46:06.145616 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:06.145580    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:46:06.185673 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:06.185637    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:06.186031 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:06.186006    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-nbf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 01 12:46:06.186243 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:06.186217    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 01 12:46:06.288344 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:06.288309    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:06.288724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:06.288702    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 01 12:46:06.288884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:06.288859    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 01 12:46:07.268606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:07.268558    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-nbf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 01 12:46:07.269233 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:07.269187    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 01 12:46:07.283396 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:07.283362    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 01 12:46:07.293506 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:07.283655    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 01 12:46:08.192515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:08.192482    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-nbf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 01 12:46:08.192812 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:08.192782    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 01 12:46:08.286459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:08.286420    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 01 12:46:08.287118 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:08.287086    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 01 12:46:09.184436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:09.184398    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-nbf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 01 12:46:09.184779 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:09.184743    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 01 12:46:09.286451 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:09.286419    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 01 12:46:09.286720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:09.286694    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 01 12:46:10.191104 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:10.184221    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-nbf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 01 12:46:10.191104 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:10.184287    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 01 12:46:10.280662 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:10.280626    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 01 12:46:10.280966 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:10.280936    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 01 12:46:11.181522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:11.181478    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-nbf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 01 12:46:11.182165 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:11.182130    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 01 12:46:11.282545 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:11.282506    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 01 12:46:11.282816 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:11.282787    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 01 12:46:12.181797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:12.181747    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-nbf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 01 12:46:12.182325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:12.181815    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 01 12:46:12.279096 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:12.279033    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 01 12:46:12.279275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:12.279122    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 01 12:46:13.181854 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:13.181223    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-nbf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 01 12:46:13.181854 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:13.181296    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 01 12:46:13.279110 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:13.279066    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 01 12:46:13.279282 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:13.279137    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 01 12:46:15.197298 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.197253    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-nbf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:46:15.197747 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.197423    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:46:15.230999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.230117    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:15.231798 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.231623    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:15.233063 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.233000    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:46:15.321234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.321190    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:46:15.321408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.321253    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:46:15.473031 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.472925    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:15.475576 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.473682    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:15.477005 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:15.476979    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:46:31.031293 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:46:31.031246    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:46:39.303141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:39.303106    5725 scope.go:122] "RemoveContainer" containerID="a3d5a8d06152da071b4a41b4b4056d4b58889b4bd2ff7ca7a0ee7e2609980cf8"
Nov 01 12:46:39.317276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:39.317246    5725 scope.go:122] "RemoveContainer" containerID="48bd65435d9decf1d5365545a197467c5912903e10482b0e54c39d25e44c70d0"
Nov 01 12:46:39.331769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:39.331730    5725 scope.go:122] "RemoveContainer" containerID="db51a2e2756847f6dee0e44fc488982f432b90685042e13a5e9bac1b30bf3092"
Nov 01 12:46:39.343496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:39.343474    5725 scope.go:122] "RemoveContainer" containerID="aed35533b27a29ef730cfc12e3995fad8e88b82706d4802924fa167a16171a3e"
Nov 01 12:46:39.354255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:39.354219    5725 scope.go:122] "RemoveContainer" containerID="6c5263653d082673a48184b596fa528db0d1ef817c27758a69b9094d4814eb50"
Nov 01 12:46:50.257576 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:50.257539    5725 generic.go:356] "Generic (PLEG): container finished" podID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerID="4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2" exitCode=0
Nov 01 12:46:50.258021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:50.257610    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" event={"ID":"69ef04e5-718b-43dc-9544-b0b4be25f9a6","Type":"ContainerDied","Data":"4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2"}
Nov 01 12:46:50.258099 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:50.258085    5725 scope.go:122] "RemoveContainer" containerID="4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2"
Nov 01 12:46:50.259798 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:50.259766    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:46:50.259955 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:50.259850    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:46:50.259955 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:50.259901    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:46:51.325445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:51.325409    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" event={"ID":"69ef04e5-718b-43dc-9544-b0b4be25f9a6","Type":"ContainerStarted","Data":"9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026"}
Nov 01 12:46:51.522694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:51.522656    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:46:51.522873 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:51.522852    5725 patch_prober.go:29] interesting pod/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 12:46:51.522919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:51.522907    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 12:46:52.523472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:52.523430    5725 patch_prober.go:29] interesting pod/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 12:46:52.523890 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:52.523492    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 12:46:53.523622 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:53.523575    5725 patch_prober.go:29] interesting pod/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 12:46:53.524040 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:53.523646    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 12:46:54.523258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:54.523224    5725 patch_prober.go:29] interesting pod/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 12:46:54.523454 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:54.523306    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 12:46:55.523878 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:55.523836    5725 patch_prober.go:29] interesting pod/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 12:46:55.524371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:55.523904    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 12:46:56.523553 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:56.523512    5725 patch_prober.go:29] interesting pod/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 01 12:46:56.523889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:56.523584    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 01 12:46:57.139093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:57.139032    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:46:57.528332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:57.528298    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:46:57.529599 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:46:57.529579    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:47:01.029593 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:47:01.029560    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:47:10.048972 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:10.048933    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"]
Nov 01 12:47:10.094388 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:10.094355    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1064198f-wmmfs"]
Nov 01 12:47:10.618425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:10.618392    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12363f69-ecc4-49c3-9633-94b5c3f01ae2" path="/var/lib/kubelet/pods/12363f69-ecc4-49c3-9633-94b5c3f01ae2/volumes"
Nov 01 12:47:11.551097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.551045    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"]
Nov 01 12:47:11.555777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.555749    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.588417 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.588387    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-78de6246-mp865\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.588604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.588572    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5cb\" (UniqueName: \"kubernetes.io/projected/696aec34-7766-4fcf-9335-b913e364cd9b-kube-api-access-5v5cb\") pod \"maas-api-revoke-keys-e2e-ait-78de6246-mp865\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.588686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.588673    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-78de6246-mp865\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.607652 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.607619    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"]
Nov 01 12:47:11.648460 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.648417    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:47:11.648706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.648452    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:47:11.690813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.690777    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-78de6246-mp865\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.691005 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.690911    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-78de6246-mp865\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.691098 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.691017    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5cb\" (UniqueName: \"kubernetes.io/projected/696aec34-7766-4fcf-9335-b913e364cd9b-kube-api-access-5v5cb\") pod \"maas-api-revoke-keys-e2e-ait-78de6246-mp865\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.691762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.691725    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-78de6246-mp865\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.693228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.693207    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-78de6246-mp865\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.744829 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.744798    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5cb\" (UniqueName: \"kubernetes.io/projected/696aec34-7766-4fcf-9335-b913e364cd9b-kube-api-access-5v5cb\") pod \"maas-api-revoke-keys-e2e-ait-78de6246-mp865\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:11.872498 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:11.872405    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:12.318561 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:12.315661    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"]
Nov 01 12:47:12.555655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:12.555606    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865" event={"ID":"696aec34-7766-4fcf-9335-b913e364cd9b","Type":"ContainerStarted","Data":"e214b0352224303e031bd94974d8163e6fed5c0286d8d137818904339df94742"}
Nov 01 12:47:13.568558 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:13.568468    5725 generic.go:356] "Generic (PLEG): container finished" podID="696aec34-7766-4fcf-9335-b913e364cd9b" containerID="fc5ff02e2f0f1eb5b28952136a92a35cd6b81e8a99982c70a4affca2034b74c8" exitCode=0
Nov 01 12:47:13.568957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:13.568557    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865" event={"ID":"696aec34-7766-4fcf-9335-b913e364cd9b","Type":"ContainerDied","Data":"fc5ff02e2f0f1eb5b28952136a92a35cd6b81e8a99982c70a4affca2034b74c8"}
Nov 01 12:47:14.789815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.789783    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:14.874137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.874057    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696aec34-7766-4fcf-9335-b913e364cd9b-kube-api-access-5v5cb\" (UniqueName: \"kubernetes.io/projected/696aec34-7766-4fcf-9335-b913e364cd9b-kube-api-access-5v5cb\") pod \"696aec34-7766-4fcf-9335-b913e364cd9b\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") "
Nov 01 12:47:14.874295 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.874149    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-openshift-ca\") pod \"696aec34-7766-4fcf-9335-b913e364cd9b\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") "
Nov 01 12:47:14.874352 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.874318    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-xks-ca\") pod \"696aec34-7766-4fcf-9335-b913e364cd9b\" (UID: \"696aec34-7766-4fcf-9335-b913e364cd9b\") "
Nov 01 12:47:14.874586 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.874557    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-openshift-ca" pod "696aec34-7766-4fcf-9335-b913e364cd9b" (UID: "696aec34-7766-4fcf-9335-b913e364cd9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:47:14.875009 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.874987    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:14.876427 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.876400    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-xks-ca" pod "696aec34-7766-4fcf-9335-b913e364cd9b" (UID: "696aec34-7766-4fcf-9335-b913e364cd9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:47:14.876514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.876448    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696aec34-7766-4fcf-9335-b913e364cd9b-kube-api-access-5v5cb" pod "696aec34-7766-4fcf-9335-b913e364cd9b" (UID: "696aec34-7766-4fcf-9335-b913e364cd9b"). InnerVolumeSpecName "kube-api-access-5v5cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:47:14.976719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.976678    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/696aec34-7766-4fcf-9335-b913e364cd9b-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:14.976719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:14.976706    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v5cb\" (UniqueName: \"kubernetes.io/projected/696aec34-7766-4fcf-9335-b913e364cd9b-kube-api-access-5v5cb\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:15.598788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.598745    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"
Nov 01 12:47:15.598788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.598764    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865" event={"ID":"696aec34-7766-4fcf-9335-b913e364cd9b","Type":"ContainerDied","Data":"e214b0352224303e031bd94974d8163e6fed5c0286d8d137818904339df94742"}
Nov 01 12:47:15.599004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.598815    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e214b0352224303e031bd94974d8163e6fed5c0286d8d137818904339df94742"
Nov 01 12:47:15.803694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.803660    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"]
Nov 01 12:47:15.804420 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.804403    5725 cpu_manager.go:414] "containerMap: removing container" podUID="696aec34-7766-4fcf-9335-b913e364cd9b" containerName="revoke-keys"
Nov 01 12:47:15.804420 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.804420    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696aec34-7766-4fcf-9335-b913e364cd9b" containerName="revoke-keys"
Nov 01 12:47:15.804556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.804529    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="696aec34-7766-4fcf-9335-b913e364cd9b" containerName="revoke-keys"
Nov 01 12:47:15.808853 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.808833    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:15.858166 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.858088    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:47:15.858342 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.858327    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:47:15.859089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.859067    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"]
Nov 01 12:47:15.890622 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.890584    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhcp\" (UniqueName: \"kubernetes.io/projected/51d7b6f3-6ede-4308-ad33-851f4ff56938-kube-api-access-8jhcp\") pod \"maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:15.890859 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.890824    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:15.890943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.890889    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:15.992691 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.992655    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhcp\" (UniqueName: \"kubernetes.io/projected/51d7b6f3-6ede-4308-ad33-851f4ff56938-kube-api-access-8jhcp\") pod \"maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:15.992893 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.992786    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:15.992893 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.992823    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:15.993486 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.993460    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:15.995196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:15.995178    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:16.047444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:16.047417    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhcp\" (UniqueName: \"kubernetes.io/projected/51d7b6f3-6ede-4308-ad33-851f4ff56938-kube-api-access-8jhcp\") pod \"maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:16.123959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:16.123854    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:16.508027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:16.507992    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"]
Nov 01 12:47:16.513171 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:47:16.512212    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d7b6f3_6ede_4308_ad33_851f4ff56938.slice/crio-72b78284bded6ac90766a539d018153ea4ea865d13b6b684721ed3a0f88332a7 WatchSource:0}: Error finding container 72b78284bded6ac90766a539d018153ea4ea865d13b6b684721ed3a0f88332a7: Status 404 returned error can't find the container with id 72b78284bded6ac90766a539d018153ea4ea865d13b6b684721ed3a0f88332a7
Nov 01 12:47:16.608484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:16.608445    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr" event={"ID":"51d7b6f3-6ede-4308-ad33-851f4ff56938","Type":"ContainerStarted","Data":"72b78284bded6ac90766a539d018153ea4ea865d13b6b684721ed3a0f88332a7"}
Nov 01 12:47:17.051527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.051490    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"]
Nov 01 12:47:17.057373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.057348    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.104918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.104843    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"]
Nov 01 12:47:17.211906 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.211868    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8n5g\" (UniqueName: \"kubernetes.io/projected/d6192132-20ed-4383-ba12-654628203624-kube-api-access-g8n5g\") pod \"maas-api-revoke-sub-034e29cb-wx5s5\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.212109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.212022    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6192132-20ed-4383-ba12-654628203624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034e29cb-wx5s5\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.212109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.212085    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6192132-20ed-4383-ba12-654628203624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034e29cb-wx5s5\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.313698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.313661    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6192132-20ed-4383-ba12-654628203624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034e29cb-wx5s5\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.313867 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.313720    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6192132-20ed-4383-ba12-654628203624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034e29cb-wx5s5\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.313931 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.313918    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8n5g\" (UniqueName: \"kubernetes.io/projected/d6192132-20ed-4383-ba12-654628203624-kube-api-access-g8n5g\") pod \"maas-api-revoke-sub-034e29cb-wx5s5\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.314429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.314405    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6192132-20ed-4383-ba12-654628203624-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034e29cb-wx5s5\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.316125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.316103    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6192132-20ed-4383-ba12-654628203624-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034e29cb-wx5s5\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.367838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.367762    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8n5g\" (UniqueName: \"kubernetes.io/projected/d6192132-20ed-4383-ba12-654628203624-kube-api-access-g8n5g\") pod \"maas-api-revoke-sub-034e29cb-wx5s5\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.374660 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.374633    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:17.618361 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.618280    5725 generic.go:356] "Generic (PLEG): container finished" podID="51d7b6f3-6ede-4308-ad33-851f4ff56938" containerID="d37002ae37ea341de9589dbeb04ce9b4c1f101b03c3a6f677b9e2d5e2d8caac9" exitCode=0
Nov 01 12:47:17.618512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.618358    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr" event={"ID":"51d7b6f3-6ede-4308-ad33-851f4ff56938","Type":"ContainerDied","Data":"d37002ae37ea341de9589dbeb04ce9b4c1f101b03c3a6f677b9e2d5e2d8caac9"}
Nov 01 12:47:17.757421 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:17.757384    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"]
Nov 01 12:47:17.761590 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:47:17.761556    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6192132_20ed_4383_ba12_654628203624.slice/crio-9b488c75a908ff9342cad321b63753e4d94d71e7592574a5fd6e71af8fe7c080 WatchSource:0}: Error finding container 9b488c75a908ff9342cad321b63753e4d94d71e7592574a5fd6e71af8fe7c080: Status 404 returned error can't find the container with id 9b488c75a908ff9342cad321b63753e4d94d71e7592574a5fd6e71af8fe7c080
Nov 01 12:47:18.016528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.016487    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"]
Nov 01 12:47:18.022618 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.022596    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.074697 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.074498    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"]
Nov 01 12:47:18.127957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.127914    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e4e-sptpd\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.128164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.127978    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7bvn\" (UniqueName: \"kubernetes.io/projected/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-kube-api-access-w7bvn\") pod \"maas-api-revoke-keys-e2e-ait-83558e4e-sptpd\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.128164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.128122    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e4e-sptpd\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.230183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.230148    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e4e-sptpd\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.230183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.230190    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7bvn\" (UniqueName: \"kubernetes.io/projected/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-kube-api-access-w7bvn\") pod \"maas-api-revoke-keys-e2e-ait-83558e4e-sptpd\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.230461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.230256    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e4e-sptpd\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.231168 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.231140    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e4e-sptpd\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.232745 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.232713    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83558e4e-sptpd\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.284228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.284192    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7bvn\" (UniqueName: \"kubernetes.io/projected/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-kube-api-access-w7bvn\") pod \"maas-api-revoke-keys-e2e-ait-83558e4e-sptpd\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.337534 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.337497    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:18.628930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.628839    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5" event={"ID":"d6192132-20ed-4383-ba12-654628203624","Type":"ContainerStarted","Data":"935c6956275c4c3f5aedb01fb0e224cecb634ddb23dbb1b4781bd69f01cd41ce"}
Nov 01 12:47:18.628930 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.628905    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5" event={"ID":"d6192132-20ed-4383-ba12-654628203624","Type":"ContainerStarted","Data":"9b488c75a908ff9342cad321b63753e4d94d71e7592574a5fd6e71af8fe7c080"}
Nov 01 12:47:18.773751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.773724    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"]
Nov 01 12:47:18.782307 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.782263    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5" podStartSLOduration=1.782248795 podStartE2EDuration="1.782248795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:47:18.776567969 +0000 UTC m=+1480.283759181" watchObservedRunningTime="2026-11-01 12:47:18.782248795 +0000 UTC m=+1480.289440011"
Nov 01 12:47:18.788134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.788104    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:18.941103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.941004    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-xks-ca\") pod \"51d7b6f3-6ede-4308-ad33-851f4ff56938\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") "
Nov 01 12:47:18.941103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.941075    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-openshift-ca\") pod \"51d7b6f3-6ede-4308-ad33-851f4ff56938\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") "
Nov 01 12:47:18.941346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.941212    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d7b6f3-6ede-4308-ad33-851f4ff56938-kube-api-access-8jhcp\" (UniqueName: \"kubernetes.io/projected/51d7b6f3-6ede-4308-ad33-851f4ff56938-kube-api-access-8jhcp\") pod \"51d7b6f3-6ede-4308-ad33-851f4ff56938\" (UID: \"51d7b6f3-6ede-4308-ad33-851f4ff56938\") "
Nov 01 12:47:18.941513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.941485    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-openshift-ca" pod "51d7b6f3-6ede-4308-ad33-851f4ff56938" (UID: "51d7b6f3-6ede-4308-ad33-851f4ff56938"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:47:18.941973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.941944    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:18.943465 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.943442    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d7b6f3-6ede-4308-ad33-851f4ff56938-kube-api-access-8jhcp" pod "51d7b6f3-6ede-4308-ad33-851f4ff56938" (UID: "51d7b6f3-6ede-4308-ad33-851f4ff56938"). InnerVolumeSpecName "kube-api-access-8jhcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:47:18.943558 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:18.943472    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-xks-ca" pod "51d7b6f3-6ede-4308-ad33-851f4ff56938" (UID: "51d7b6f3-6ede-4308-ad33-851f4ff56938"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:47:19.043644 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.043619    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d7b6f3-6ede-4308-ad33-851f4ff56938-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:19.043771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.043646    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jhcp\" (UniqueName: \"kubernetes.io/projected/51d7b6f3-6ede-4308-ad33-851f4ff56938-kube-api-access-8jhcp\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:19.640631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.640598    5725 generic.go:356] "Generic (PLEG): container finished" podID="d6192132-20ed-4383-ba12-654628203624" containerID="935c6956275c4c3f5aedb01fb0e224cecb634ddb23dbb1b4781bd69f01cd41ce" exitCode=0
Nov 01 12:47:19.641097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.640685    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5" event={"ID":"d6192132-20ed-4383-ba12-654628203624","Type":"ContainerDied","Data":"935c6956275c4c3f5aedb01fb0e224cecb634ddb23dbb1b4781bd69f01cd41ce"}
Nov 01 12:47:19.642402 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.642380    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd" event={"ID":"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e","Type":"ContainerStarted","Data":"80e3dba2b01c6d5a8be9cfd3722db5bdc50886f9a7189225b997efad7c54d3ec"}
Nov 01 12:47:19.642485 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.642423    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd" event={"ID":"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e","Type":"ContainerStarted","Data":"0e54110a80f246198073af12551b387cd24d5a07abf1023f1df2db3f3d953ff2"}
Nov 01 12:47:19.643987 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.643967    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"
Nov 01 12:47:19.644155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.643979    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr" event={"ID":"51d7b6f3-6ede-4308-ad33-851f4ff56938","Type":"ContainerDied","Data":"72b78284bded6ac90766a539d018153ea4ea865d13b6b684721ed3a0f88332a7"}
Nov 01 12:47:19.644155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.644029    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72b78284bded6ac90766a539d018153ea4ea865d13b6b684721ed3a0f88332a7"
Nov 01 12:47:19.910315 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:19.910214    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd" podStartSLOduration=2.910200285 podStartE2EDuration="2.910200285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:47:19.906693408 +0000 UTC m=+1481.413884636" watchObservedRunningTime="2026-11-01 12:47:19.910200285 +0000 UTC m=+1481.417391501"
Nov 01 12:47:20.654775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.654739    5725 generic.go:356] "Generic (PLEG): container finished" podID="a5ebe2b3-0662-43c3-8143-cd7d69b03f7e" containerID="80e3dba2b01c6d5a8be9cfd3722db5bdc50886f9a7189225b997efad7c54d3ec" exitCode=0
Nov 01 12:47:20.655220 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.654823    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd" event={"ID":"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e","Type":"ContainerDied","Data":"80e3dba2b01c6d5a8be9cfd3722db5bdc50886f9a7189225b997efad7c54d3ec"}
Nov 01 12:47:20.836714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.836683    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:20.976346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.976311    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6192132-20ed-4383-ba12-654628203624-kube-api-access-g8n5g\" (UniqueName: \"kubernetes.io/projected/d6192132-20ed-4383-ba12-654628203624-kube-api-access-g8n5g\") pod \"d6192132-20ed-4383-ba12-654628203624\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") "
Nov 01 12:47:20.976514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.976435    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6192132-20ed-4383-ba12-654628203624-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6192132-20ed-4383-ba12-654628203624-maas-api-xks-ca\") pod \"d6192132-20ed-4383-ba12-654628203624\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") "
Nov 01 12:47:20.976730 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.976705    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6192132-20ed-4383-ba12-654628203624-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6192132-20ed-4383-ba12-654628203624-maas-api-openshift-ca\") pod \"d6192132-20ed-4383-ba12-654628203624\" (UID: \"d6192132-20ed-4383-ba12-654628203624\") "
Nov 01 12:47:20.977113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.977091    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6192132-20ed-4383-ba12-654628203624-maas-api-openshift-ca" pod "d6192132-20ed-4383-ba12-654628203624" (UID: "d6192132-20ed-4383-ba12-654628203624"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:47:20.977609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.977588    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6192132-20ed-4383-ba12-654628203624-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:20.978537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.978515    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6192132-20ed-4383-ba12-654628203624-maas-api-xks-ca" pod "d6192132-20ed-4383-ba12-654628203624" (UID: "d6192132-20ed-4383-ba12-654628203624"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:47:20.978620 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:20.978599    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6192132-20ed-4383-ba12-654628203624-kube-api-access-g8n5g" pod "d6192132-20ed-4383-ba12-654628203624" (UID: "d6192132-20ed-4383-ba12-654628203624"). InnerVolumeSpecName "kube-api-access-g8n5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:47:21.079393 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.079353    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6192132-20ed-4383-ba12-654628203624-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:21.079393 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.079391    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8n5g\" (UniqueName: \"kubernetes.io/projected/d6192132-20ed-4383-ba12-654628203624-kube-api-access-g8n5g\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:21.668286 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.668247    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5" event={"ID":"d6192132-20ed-4383-ba12-654628203624","Type":"ContainerDied","Data":"9b488c75a908ff9342cad321b63753e4d94d71e7592574a5fd6e71af8fe7c080"}
Nov 01 12:47:21.668763 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.668315    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b488c75a908ff9342cad321b63753e4d94d71e7592574a5fd6e71af8fe7c080"
Nov 01 12:47:21.668763 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.668257    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"
Nov 01 12:47:21.858254 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.858229    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:21.892258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.892224    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-xks-ca\") pod \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") "
Nov 01 12:47:21.892494 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.892477    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-openshift-ca\") pod \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") "
Nov 01 12:47:21.892602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.892586    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-kube-api-access-w7bvn\" (UniqueName: \"kubernetes.io/projected/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-kube-api-access-w7bvn\") pod \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\" (UID: \"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e\") "
Nov 01 12:47:21.892848 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.892821    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-openshift-ca" pod "a5ebe2b3-0662-43c3-8143-cd7d69b03f7e" (UID: "a5ebe2b3-0662-43c3-8143-cd7d69b03f7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:47:21.893135 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.893117    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:21.894295 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.894272    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-xks-ca" pod "a5ebe2b3-0662-43c3-8143-cd7d69b03f7e" (UID: "a5ebe2b3-0662-43c3-8143-cd7d69b03f7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:47:21.894578 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.894552    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-kube-api-access-w7bvn" pod "a5ebe2b3-0662-43c3-8143-cd7d69b03f7e" (UID: "a5ebe2b3-0662-43c3-8143-cd7d69b03f7e"). InnerVolumeSpecName "kube-api-access-w7bvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:47:21.994836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.994800    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7bvn\" (UniqueName: \"kubernetes.io/projected/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-kube-api-access-w7bvn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:21.994836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:21.994838    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:47:22.679087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:22.679039    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd" event={"ID":"a5ebe2b3-0662-43c3-8143-cd7d69b03f7e","Type":"ContainerDied","Data":"0e54110a80f246198073af12551b387cd24d5a07abf1023f1df2db3f3d953ff2"}
Nov 01 12:47:22.679087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:22.679042    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"
Nov 01 12:47:22.679496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:22.679108    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e54110a80f246198073af12551b387cd24d5a07abf1023f1df2db3f3d953ff2"
Nov 01 12:47:31.033084 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:47:31.031332    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:47:38.052039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:38.052002    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"]
Nov 01 12:47:38.096530 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:38.096493    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb5da01-hj5sb"]
Nov 01 12:47:38.620015 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:38.619981    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22225b05-f5e3-4911-9240-38c7431eb4a9" path="/var/lib/kubelet/pods/22225b05-f5e3-4911-9240-38c7431eb4a9/volumes"
Nov 01 12:47:38.978431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:38.978405    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:47:38.981459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:38.981436    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:47:38.986946 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:38.986923    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:47:38.990058 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:38.990031    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:47:40.143082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.143036    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"]
Nov 01 12:47:40.143969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.143946    5725 cpu_manager.go:414] "containerMap: removing container" podUID="d6192132-20ed-4383-ba12-654628203624" containerName="revoke-keys"
Nov 01 12:47:40.143969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.143969    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6192132-20ed-4383-ba12-654628203624" containerName="revoke-keys"
Nov 01 12:47:40.144154 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.144001    5725 cpu_manager.go:414] "containerMap: removing container" podUID="51d7b6f3-6ede-4308-ad33-851f4ff56938" containerName="revoke-keys"
Nov 01 12:47:40.144154 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.144008    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d7b6f3-6ede-4308-ad33-851f4ff56938" containerName="revoke-keys"
Nov 01 12:47:40.144154 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.144031    5725 cpu_manager.go:414] "containerMap: removing container" podUID="a5ebe2b3-0662-43c3-8143-cd7d69b03f7e" containerName="revoke-keys"
Nov 01 12:47:40.144154 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.144038    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ebe2b3-0662-43c3-8143-cd7d69b03f7e" containerName="revoke-keys"
Nov 01 12:47:40.144369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.144228    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6192132-20ed-4383-ba12-654628203624" containerName="revoke-keys"
Nov 01 12:47:40.144369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.144248    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d7b6f3-6ede-4308-ad33-851f4ff56938" containerName="revoke-keys"
Nov 01 12:47:40.144369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.144260    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ebe2b3-0662-43c3-8143-cd7d69b03f7e" containerName="revoke-keys"
Nov 01 12:47:40.148607 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.148587    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.196026 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.193370    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-envoy\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.196334 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.196311    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-token\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.196470 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.196456    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-data\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.196675 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.196647    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-podinfo\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.196843 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.196816    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-socket\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.207514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.207482    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-certs\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.207883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.207860    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fc942-dd8e-41df-8754-55ae9ab9cf46-istiod-ca-cert\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.208035 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.208016    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-credential-socket\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.208257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.208238    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxlhw\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-kube-api-access-hxlhw\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.255078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.254415    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-6c650b-openshift-default-dockercfg-kmbw7\""
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.360773    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-certs\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.360922    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fc942-dd8e-41df-8754-55ae9ab9cf46-istiod-ca-cert\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.360965    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-credential-socket\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.361042    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxlhw\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-kube-api-access-hxlhw\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.361159    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-envoy\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.361213    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-token\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.361245    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-data\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.361325    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-podinfo\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.361388    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-socket\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.362012    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-socket\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.362325    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-certs\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.362982    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fc942-dd8e-41df-8754-55ae9ab9cf46-istiod-ca-cert\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.444213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.418501    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-data\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.445390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.445356    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-credential-socket\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.536113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.528775    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxlhw\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-kube-api-access-hxlhw\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.602789 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.584195    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"]
Nov 01 12:47:40.614851 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.608370    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-token\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.622245 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.622148    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-envoy\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.629270 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.629239    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-podinfo\") pod \"e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") " pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:40.800508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:40.800461    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:41.282285 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:41.282253    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"]
Nov 01 12:47:41.282648 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:47:41.282389    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800fc942_dd8e_41df_8754_55ae9ab9cf46.slice/crio-b9443dc3b35550270e955b1f997917f31d4f1fbf1b8011df114d261f95f5bfbb WatchSource:0}: Error finding container b9443dc3b35550270e955b1f997917f31d4f1fbf1b8011df114d261f95f5bfbb: Status 404 returned error can't find the container with id b9443dc3b35550270e955b1f997917f31d4f1fbf1b8011df114d261f95f5bfbb
Nov 01 12:47:41.285354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:41.285288    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:47:41.285452 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:41.285405    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:47:41.285452 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:41.285447    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:47:41.980151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:41.980106    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" event={"ID":"800fc942-dd8e-41df-8754-55ae9ab9cf46","Type":"ContainerStarted","Data":"04758c8923fdf57090c3dec06241d0b93ccd419606908b7f3be00ff8a16f4406"}
Nov 01 12:47:41.980354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:41.980178    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" event={"ID":"800fc942-dd8e-41df-8754-55ae9ab9cf46","Type":"ContainerStarted","Data":"b9443dc3b35550270e955b1f997917f31d4f1fbf1b8011df114d261f95f5bfbb"}
Nov 01 12:47:42.191388 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:42.191333    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" podStartSLOduration=2.191317806 podStartE2EDuration="2.191317806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:47:42.139721477 +0000 UTC m=+1503.646912695" watchObservedRunningTime="2026-11-01 12:47:42.191317806 +0000 UTC m=+1503.698509023"
Nov 01 12:47:42.801169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:42.801130    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:42.807131 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:42.807105    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:42.993338 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:42.993292    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:42.994538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:42.994513    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:47:55.052833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:55.052795    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"]
Nov 01 12:47:55.096246 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:55.096211    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235855b-5kbnx"]
Nov 01 12:47:56.621097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:47:56.621040    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5848c802-b889-40d6-a7e9-2cc485e01847" path="/var/lib/kubelet/pods/5848c802-b889-40d6-a7e9-2cc485e01847/volumes"
Nov 01 12:48:01.219477 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:48:01.219386    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:48:01.517237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:01.517200    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-sscmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:48:01.517567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:01.517534    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:48:05.610666 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:05.610623    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"]
Nov 01 12:48:05.715137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:05.714373    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78de6246-mp865"]
Nov 01 12:48:06.618108 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:06.618066    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696aec34-7766-4fcf-9335-b913e364cd9b" path="/var/lib/kubelet/pods/696aec34-7766-4fcf-9335-b913e364cd9b/volumes"
Nov 01 12:48:07.496287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:07.496111    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"]
Nov 01 12:48:07.616453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:07.616399    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83558e4e-sptpd"]
Nov 01 12:48:08.621433 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:08.621398    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ebe2b3-0662-43c3-8143-cd7d69b03f7e" path="/var/lib/kubelet/pods/a5ebe2b3-0662-43c3-8143-cd7d69b03f7e/volumes"
Nov 01 12:48:09.268855 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.268815    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"]
Nov 01 12:48:09.275107 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.275080    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.324801 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.324767    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:48:09.400417 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.400376    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:48:09.408932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.408895    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"]
Nov 01 12:48:09.417567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.417533    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jw4\" (UniqueName: \"kubernetes.io/projected/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-kube-api-access-k9jw4\") pod \"maas-api-revoke-sub-9690124c-jl968\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.417804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.417582    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9690124c-jl968\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.417804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.417714    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9690124c-jl968\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.518947 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.518846    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jw4\" (UniqueName: \"kubernetes.io/projected/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-kube-api-access-k9jw4\") pod \"maas-api-revoke-sub-9690124c-jl968\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.518947 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.518894    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9690124c-jl968\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.519234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.518977    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9690124c-jl968\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.519891 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.519850    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9690124c-jl968\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.521809 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.521785    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9690124c-jl968\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.581715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.581680    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jw4\" (UniqueName: \"kubernetes.io/projected/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-kube-api-access-k9jw4\") pod \"maas-api-revoke-sub-9690124c-jl968\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:09.590643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:09.590610    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:10.481300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:10.478409    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"]
Nov 01 12:48:10.586617 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:10.586574    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968" event={"ID":"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa","Type":"ContainerStarted","Data":"4a32f6e5e15debcc939bc93f2502c92e7209c1e5472f5d2e2ee4293d7330c753"}
Nov 01 12:48:11.599026 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:11.598992    5725 generic.go:356] "Generic (PLEG): container finished" podID="74866cb4-c8da-49fe-a0d0-6f46ea8a87fa" containerID="67a40b77a952337fd00e9d45a00aadb59f7e4d443d75abdfc8e5b82c3fdc5815" exitCode=0
Nov 01 12:48:11.599493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:11.599106    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968" event={"ID":"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa","Type":"ContainerDied","Data":"67a40b77a952337fd00e9d45a00aadb59f7e4d443d75abdfc8e5b82c3fdc5815"}
Nov 01 12:48:12.983193 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:12.982940    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:13.128069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.128011    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-openshift-ca\") pod \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") "
Nov 01 12:48:13.132705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.128673    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-openshift-ca" pod "74866cb4-c8da-49fe-a0d0-6f46ea8a87fa" (UID: "74866cb4-c8da-49fe-a0d0-6f46ea8a87fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:48:13.133917 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.133895    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-xks-ca\") pod \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") "
Nov 01 12:48:13.134156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.134131    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-kube-api-access-k9jw4\" (UniqueName: \"kubernetes.io/projected/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-kube-api-access-k9jw4\") pod \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\" (UID: \"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa\") "
Nov 01 12:48:13.135994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.135973    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:13.181307 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.181226    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-xks-ca" pod "74866cb4-c8da-49fe-a0d0-6f46ea8a87fa" (UID: "74866cb4-c8da-49fe-a0d0-6f46ea8a87fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:48:13.195083 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.189768    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-kube-api-access-k9jw4" pod "74866cb4-c8da-49fe-a0d0-6f46ea8a87fa" (UID: "74866cb4-c8da-49fe-a0d0-6f46ea8a87fa"). InnerVolumeSpecName "kube-api-access-k9jw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:48:13.269375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.258309    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:13.285158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.285123    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9jw4\" (UniqueName: \"kubernetes.io/projected/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa-kube-api-access-k9jw4\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:13.635399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.635318    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"
Nov 01 12:48:13.635399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.635354    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968" event={"ID":"74866cb4-c8da-49fe-a0d0-6f46ea8a87fa","Type":"ContainerDied","Data":"4a32f6e5e15debcc939bc93f2502c92e7209c1e5472f5d2e2ee4293d7330c753"}
Nov 01 12:48:13.635589 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:13.635411    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a32f6e5e15debcc939bc93f2502c92e7209c1e5472f5d2e2ee4293d7330c753"
Nov 01 12:48:16.364028 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:16.363997    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"]
Nov 01 12:48:16.416344 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:16.416201    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ccbb55-pkqxr"]
Nov 01 12:48:16.427899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:16.427865    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"]
Nov 01 12:48:16.428189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:16.428158    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy" containerID="cri-o://9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026" gracePeriod=30
Nov 01 12:48:16.657739 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:16.657646    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d7b6f3-6ede-4308-ad33-851f4ff56938" path="/var/lib/kubelet/pods/51d7b6f3-6ede-4308-ad33-851f4ff56938/volumes"
Nov 01 12:48:21.074031 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.073772    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"]
Nov 01 12:48:21.101814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.101761    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb64dbf-xll6l"]
Nov 01 12:48:21.723815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.723791    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:48:21.762689 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.762657    5725 generic.go:356] "Generic (PLEG): container finished" podID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerID="9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026" exitCode=0
Nov 01 12:48:21.762868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.762732    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"
Nov 01 12:48:21.762868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.762735    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" event={"ID":"69ef04e5-718b-43dc-9544-b0b4be25f9a6","Type":"ContainerDied","Data":"9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026"}
Nov 01 12:48:21.762868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.762815    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9" event={"ID":"69ef04e5-718b-43dc-9544-b0b4be25f9a6","Type":"ContainerDied","Data":"8645dcb6649024c07b025dd34af34bea7b2afef6172ab51af4ba2e7f5c0d97d9"}
Nov 01 12:48:21.762868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.762843    5725 scope.go:122] "RemoveContainer" containerID="9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026"
Nov 01 12:48:21.775693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.775667    5725 scope.go:122] "RemoveContainer" containerID="4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2"
Nov 01 12:48:21.788939 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.788917    5725 scope.go:122] "RemoveContainer" containerID="9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026"
Nov 01 12:48:21.789309 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:48:21.789281    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026\": container with ID starting with 9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026 not found: ID does not exist" containerID="9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026"
Nov 01 12:48:21.789381 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.789321    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026"} err="failed to get container status \"9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026\": rpc error: code = NotFound desc = could not find container \"9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026\": container with ID starting with 9c072f307fce8251c193acc23b4bee1566c0a0281f83acb197265ff36b72f026 not found: ID does not exist"
Nov 01 12:48:21.789381 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.789347    5725 scope.go:122] "RemoveContainer" containerID="4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2"
Nov 01 12:48:21.789631 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:48:21.789610    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2\": container with ID starting with 4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2 not found: ID does not exist" containerID="4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2"
Nov 01 12:48:21.789765 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.789633    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2"} err="failed to get container status \"4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2\": rpc error: code = NotFound desc = could not find container \"4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2\": container with ID starting with 4fcb1662d13d8a1736ae4c4f9b1cd08a9e4685e847161ceddccc43b9395cfaf2 not found: ID does not exist"
Nov 01 12:48:21.832395 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832354    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istiod-ca-cert\") pod \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") "
Nov 01 12:48:21.832555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832444    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-envoy\") pod \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") "
Nov 01 12:48:21.832555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832538    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-token\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-token\") pod \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") "
Nov 01 12:48:21.832688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832573    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-data\") pod \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") "
Nov 01 12:48:21.832688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832626    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-credential-socket\") pod \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") "
Nov 01 12:48:21.832814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832719    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-socket\") pod \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") "
Nov 01 12:48:21.832814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832767    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-kube-api-access-bhr89\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-kube-api-access-bhr89\") pod \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") "
Nov 01 12:48:21.832923 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832851    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-data" pod "69ef04e5-718b-43dc-9544-b0b4be25f9a6" (UID: "69ef04e5-718b-43dc-9544-b0b4be25f9a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:48:21.832982 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832951    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-podinfo\") pod \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") "
Nov 01 12:48:21.833074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.832997    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-certs\") pod \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\" (UID: \"69ef04e5-718b-43dc-9544-b0b4be25f9a6\") "
Nov 01 12:48:21.833150 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.833034    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istiod-ca-cert" pod "69ef04e5-718b-43dc-9544-b0b4be25f9a6" (UID: "69ef04e5-718b-43dc-9544-b0b4be25f9a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:48:21.833205 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.833147    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-credential-socket" pod "69ef04e5-718b-43dc-9544-b0b4be25f9a6" (UID: "69ef04e5-718b-43dc-9544-b0b4be25f9a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:48:21.833559 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.833520    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-socket" pod "69ef04e5-718b-43dc-9544-b0b4be25f9a6" (UID: "69ef04e5-718b-43dc-9544-b0b4be25f9a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:48:21.833792 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.833763    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-certs" pod "69ef04e5-718b-43dc-9544-b0b4be25f9a6" (UID: "69ef04e5-718b-43dc-9544-b0b4be25f9a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:48:21.834198 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.834175    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:21.834332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.834202    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:21.834332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.834217    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:21.834332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.834232    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:21.834332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.834245    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:21.835246 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.835218    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-token" pod "69ef04e5-718b-43dc-9544-b0b4be25f9a6" (UID: "69ef04e5-718b-43dc-9544-b0b4be25f9a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:48:21.835331 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.835240    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-envoy" pod "69ef04e5-718b-43dc-9544-b0b4be25f9a6" (UID: "69ef04e5-718b-43dc-9544-b0b4be25f9a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:48:21.835331 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.835312    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-podinfo" pod "69ef04e5-718b-43dc-9544-b0b4be25f9a6" (UID: "69ef04e5-718b-43dc-9544-b0b4be25f9a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:48:21.835601 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.835582    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-kube-api-access-bhr89" pod "69ef04e5-718b-43dc-9544-b0b4be25f9a6" (UID: "69ef04e5-718b-43dc-9544-b0b4be25f9a6"). InnerVolumeSpecName "kube-api-access-bhr89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:48:21.936068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.936026    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:21.936068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.936072    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:21.936301 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.936095    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhr89\" (UniqueName: \"kubernetes.io/projected/69ef04e5-718b-43dc-9544-b0b4be25f9a6-kube-api-access-bhr89\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:21.936301 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:21.936106    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ef04e5-718b-43dc-9544-b0b4be25f9a6-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:22.185724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:22.185691    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"]
Nov 01 12:48:22.190882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:22.190850    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b2ccbb55-openshift-default-565f4f6f69-qb5r9"]
Nov 01 12:48:22.617631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:22.617598    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" path="/var/lib/kubelet/pods/69ef04e5-718b-43dc-9544-b0b4be25f9a6/volumes"
Nov 01 12:48:22.618136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:22.618121    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f096267-c0f2-4ef1-b356-5fc51bdc8602" path="/var/lib/kubelet/pods/8f096267-c0f2-4ef1-b356-5fc51bdc8602/volumes"
Nov 01 12:48:28.491902 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.491862    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"]
Nov 01 12:48:28.492687 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.492669    5725 cpu_manager.go:414] "containerMap: removing container" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy"
Nov 01 12:48:28.492738 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.492689    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy"
Nov 01 12:48:28.492738 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.492709    5725 cpu_manager.go:414] "containerMap: removing container" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy"
Nov 01 12:48:28.492738 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.492714    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy"
Nov 01 12:48:28.492738 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.492726    5725 cpu_manager.go:414] "containerMap: removing container" podUID="74866cb4-c8da-49fe-a0d0-6f46ea8a87fa" containerName="revoke-keys"
Nov 01 12:48:28.492738 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.492732    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74866cb4-c8da-49fe-a0d0-6f46ea8a87fa" containerName="revoke-keys"
Nov 01 12:48:28.492897 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.492841    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy"
Nov 01 12:48:28.492897 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.492854    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ef04e5-718b-43dc-9544-b0b4be25f9a6" containerName="istio-proxy"
Nov 01 12:48:28.492897 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.492861    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="74866cb4-c8da-49fe-a0d0-6f46ea8a87fa" containerName="revoke-keys"
Nov 01 12:48:28.496665 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.496648    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.546399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.546351    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 12:48:28.546399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.546375    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 12:48:28.546399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.546392    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mmr7k\""
Nov 01 12:48:28.546399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.546380    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 12:48:28.602529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.602490    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"]
Nov 01 12:48:28.641370 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.641317    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.641586 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.641446    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmb97\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-kube-api-access-nmb97\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.641586 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.641503    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.641726 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.641672    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.641726 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.641719    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908e5730-2323-4fbc-a1b2-f408979ee787-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.641840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.641792    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.641840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.641833    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.743170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.743090    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmb97\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-kube-api-access-nmb97\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.743170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.743143    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.743405 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.743270    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.743405 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.743305    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908e5730-2323-4fbc-a1b2-f408979ee787-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.743405 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.743362    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.743405 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.743389    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.743649 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.743435    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.744072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.744029    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.746151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.746124    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.746298 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.746276    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908e5730-2323-4fbc-a1b2-f408979ee787-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.746364 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.746348    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.746461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.746439    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.800077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.800010    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.800234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.800220    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmb97\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-kube-api-access-nmb97\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7zz2j\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:28.810790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:28.810763    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:29.256083 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:29.256014    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"]
Nov 01 12:48:29.260875 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:48:29.260833    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908e5730_2323_4fbc_a1b2_f408979ee787.slice/crio-88733a2e8f33f59dc25c49f92911e10af3d9d3417a36e2fc4fb3fdb737862a08 WatchSource:0}: Error finding container 88733a2e8f33f59dc25c49f92911e10af3d9d3417a36e2fc4fb3fdb737862a08: Status 404 returned error can't find the container with id 88733a2e8f33f59dc25c49f92911e10af3d9d3417a36e2fc4fb3fdb737862a08
Nov 01 12:48:29.844281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:29.844238    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j" event={"ID":"908e5730-2323-4fbc-a1b2-f408979ee787","Type":"ContainerStarted","Data":"88733a2e8f33f59dc25c49f92911e10af3d9d3417a36e2fc4fb3fdb737862a08"}
Nov 01 12:48:31.030329 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:48:31.030290    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:48:33.395662 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:33.395619    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:48:33.395999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:33.395708    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:48:33.895189 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:33.895142    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j" event={"ID":"908e5730-2323-4fbc-a1b2-f408979ee787","Type":"ContainerStarted","Data":"7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4"}
Nov 01 12:48:33.895403 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:33.895209    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:34.105622 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:34.105569    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j" podStartSLOduration=1.974536728 podStartE2EDuration="6.105553453s" totalImagesPullingTime="4.131016725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:48:34.100760627 +0000 UTC m=+1555.607951868" watchObservedRunningTime="2026-11-01 12:48:34.105553453 +0000 UTC m=+1555.612744669"
Nov 01 12:48:34.924676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:34.924202    5725 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7zz2j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:48:34.924676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:34.924262    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j" podUID="908e5730-2323-4fbc-a1b2-f408979ee787" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:48:35.053977 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:35.053926    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"]
Nov 01 12:48:35.125088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:35.123525    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed61bd-jbllh"]
Nov 01 12:48:36.155202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:36.155160    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:48:36.155837 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:36.155803    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:48:36.635721 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:36.635198    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e839a2-97e8-490f-9de6-98349829b929" path="/var/lib/kubelet/pods/28e839a2-97e8-490f-9de6-98349829b929/volumes"
Nov 01 12:48:37.972099 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:37.972068    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:48:39.430988 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:39.430949    5725 scope.go:122] "RemoveContainer" containerID="e4531fe4e420b961ff3f64f1ba44b2445ebb37c9f2acd58ffd415b39a4092595"
Nov 01 12:48:39.443066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:39.443031    5725 scope.go:122] "RemoveContainer" containerID="323b404755ba397de73a453fa62c035c3edc27806824ed0813624a9848f1735d"
Nov 01 12:48:47.050158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:47.050117    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"]
Nov 01 12:48:47.097021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:47.096982    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d4d6e8d-l2smg"]
Nov 01 12:48:48.618744 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:48.618707    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a213c0-bc4e-4e52-b254-0e0776a0290f" path="/var/lib/kubelet/pods/33a213c0-bc4e-4e52-b254-0e0776a0290f/volumes"
Nov 01 12:48:54.056927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:54.056878    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"]
Nov 01 12:48:54.096684 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:54.096636    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30136c3e-gvk22"]
Nov 01 12:48:54.618105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:54.618062    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ea3116-f940-48fc-9f4b-811dab290f3d" path="/var/lib/kubelet/pods/08ea3116-f940-48fc-9f4b-811dab290f3d/volumes"
Nov 01 12:48:59.210407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:59.210106    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"]
Nov 01 12:48:59.315291 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:48:59.315246    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e147669-m6mx9"]
Nov 01 12:49:00.049744 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:00.049699    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"]
Nov 01 12:49:00.098808 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:00.098767    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9e061-dljrn"]
Nov 01 12:49:00.619606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:00.619571    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf2c770-a555-43bc-bbcf-3c6beb11d60b" path="/var/lib/kubelet/pods/2cf2c770-a555-43bc-bbcf-3c6beb11d60b/volumes"
Nov 01 12:49:00.620132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:00.620117    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8558ba68-b862-4810-8e98-970ea9decba4" path="/var/lib/kubelet/pods/8558ba68-b862-4810-8e98-970ea9decba4/volumes"
Nov 01 12:49:01.033332 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:49:01.033291    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:49:02.047781 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:02.047719    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"]
Nov 01 12:49:02.093711 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:02.093676    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9882a575-bppwz"]
Nov 01 12:49:02.619423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:02.619373    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96357a3-b6af-4d20-b0f1-c39472546a60" path="/var/lib/kubelet/pods/d96357a3-b6af-4d20-b0f1-c39472546a60/volumes"
Nov 01 12:49:04.055745 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:04.055708    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"]
Nov 01 12:49:04.102784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:04.102748    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0111d7-f7mtt"]
Nov 01 12:49:04.621363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:04.621324    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b947bb64-ac64-42ea-9aac-3b4ae73d1e9a" path="/var/lib/kubelet/pods/b947bb64-ac64-42ea-9aac-3b4ae73d1e9a/volumes"
Nov 01 12:49:09.052349 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:09.052309    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"]
Nov 01 12:49:09.095898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:09.095853    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91d70d47-jfv6g"]
Nov 01 12:49:10.619802 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:10.619765    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c576d1c-8cab-4947-b614-c4159156994e" path="/var/lib/kubelet/pods/0c576d1c-8cab-4947-b614-c4159156994e/volumes"
Nov 01 12:49:17.053841 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:17.053798    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"]
Nov 01 12:49:17.094609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:17.094573    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9774a4a8-ln7q6"]
Nov 01 12:49:18.627947 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:18.627905    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7cecc5-5b8c-4d88-9b9e-220ee75730c0" path="/var/lib/kubelet/pods/0c7cecc5-5b8c-4d88-9b9e-220ee75730c0/volumes"
Nov 01 12:49:18.630140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:18.630116    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"]
Nov 01 12:49:18.630540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:18.630511    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy" containerID="cri-o://5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a" gracePeriod=30
Nov 01 12:49:23.492867 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:23.492831    5725 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:49:23.493344 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:23.492893    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:49:23.924650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:23.924578    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:49:24.016066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.016023    5725 generic.go:356] "Generic (PLEG): container finished" podID="04995d67-9144-429c-b655-0c1fd37ad11c" containerID="5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a" exitCode=0
Nov 01 12:49:24.016269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.016099    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"
Nov 01 12:49:24.016269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.016090    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" event={"ID":"04995d67-9144-429c-b655-0c1fd37ad11c","Type":"ContainerDied","Data":"5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a"}
Nov 01 12:49:24.016269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.016226    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb" event={"ID":"04995d67-9144-429c-b655-0c1fd37ad11c","Type":"ContainerDied","Data":"6a7656e9114535d1f300cde6ac1fe987d58dc7d315595e4e69b55e5ebb52cf68"}
Nov 01 12:49:24.016269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.016255    5725 scope.go:122] "RemoveContainer" containerID="5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a"
Nov 01 12:49:24.029609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.029586    5725 scope.go:122] "RemoveContainer" containerID="5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a"
Nov 01 12:49:24.030001 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:49:24.029976    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a\": container with ID starting with 5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a not found: ID does not exist" containerID="5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a"
Nov 01 12:49:24.030128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.030015    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a"} err="failed to get container status \"5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a\": rpc error: code = NotFound desc = could not find container \"5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a\": container with ID starting with 5fcccfa9dc20d03f6d2d59a16f1ec9523fd684fc525950e08159625a1d65c97a not found: ID does not exist"
Nov 01 12:49:24.123171 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123128    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-data\") pod \"04995d67-9144-429c-b655-0c1fd37ad11c\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") "
Nov 01 12:49:24.123364 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123182    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-certs\") pod \"04995d67-9144-429c-b655-0c1fd37ad11c\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") "
Nov 01 12:49:24.123364 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123243    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-credential-socket\") pod \"04995d67-9144-429c-b655-0c1fd37ad11c\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") "
Nov 01 12:49:24.123613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123428    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04995d67-9144-429c-b655-0c1fd37ad11c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04995d67-9144-429c-b655-0c1fd37ad11c-istiod-ca-cert\") pod \"04995d67-9144-429c-b655-0c1fd37ad11c\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") "
Nov 01 12:49:24.123613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123448    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-data" pod "04995d67-9144-429c-b655-0c1fd37ad11c" (UID: "04995d67-9144-429c-b655-0c1fd37ad11c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:49:24.123613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123459    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-certs" pod "04995d67-9144-429c-b655-0c1fd37ad11c" (UID: "04995d67-9144-429c-b655-0c1fd37ad11c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:49:24.123613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123486    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-envoy\") pod \"04995d67-9144-429c-b655-0c1fd37ad11c\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") "
Nov 01 12:49:24.123613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123522    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04995d67-9144-429c-b655-0c1fd37ad11c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04995d67-9144-429c-b655-0c1fd37ad11c-istio-podinfo\") pod \"04995d67-9144-429c-b655-0c1fd37ad11c\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") "
Nov 01 12:49:24.123613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123569    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-kube-api-access-zdpqr\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-kube-api-access-zdpqr\") pod \"04995d67-9144-429c-b655-0c1fd37ad11c\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") "
Nov 01 12:49:24.123957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123634    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-istio-token\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-istio-token\") pod \"04995d67-9144-429c-b655-0c1fd37ad11c\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") "
Nov 01 12:49:24.123957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123696    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-socket\") pod \"04995d67-9144-429c-b655-0c1fd37ad11c\" (UID: \"04995d67-9144-429c-b655-0c1fd37ad11c\") "
Nov 01 12:49:24.123957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.123823    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04995d67-9144-429c-b655-0c1fd37ad11c-istiod-ca-cert" pod "04995d67-9144-429c-b655-0c1fd37ad11c" (UID: "04995d67-9144-429c-b655-0c1fd37ad11c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:49:24.124277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.124244    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-credential-socket" pod "04995d67-9144-429c-b655-0c1fd37ad11c" (UID: "04995d67-9144-429c-b655-0c1fd37ad11c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:49:24.124555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.124508    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-socket" pod "04995d67-9144-429c-b655-0c1fd37ad11c" (UID: "04995d67-9144-429c-b655-0c1fd37ad11c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:49:24.124650 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.124560    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:49:24.124755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.124644    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:49:24.124755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.124665    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:49:24.124755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.124679    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04995d67-9144-429c-b655-0c1fd37ad11c-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:49:24.126747 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.126721    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04995d67-9144-429c-b655-0c1fd37ad11c-istio-podinfo" pod "04995d67-9144-429c-b655-0c1fd37ad11c" (UID: "04995d67-9144-429c-b655-0c1fd37ad11c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:49:24.126979 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.126940    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-kube-api-access-zdpqr" pod "04995d67-9144-429c-b655-0c1fd37ad11c" (UID: "04995d67-9144-429c-b655-0c1fd37ad11c"). InnerVolumeSpecName "kube-api-access-zdpqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:49:24.128886 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.128645    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-envoy" pod "04995d67-9144-429c-b655-0c1fd37ad11c" (UID: "04995d67-9144-429c-b655-0c1fd37ad11c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:49:24.129299 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.129272    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-istio-token" pod "04995d67-9144-429c-b655-0c1fd37ad11c" (UID: "04995d67-9144-429c-b655-0c1fd37ad11c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:49:24.226513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.226479    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:49:24.226513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.226511    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04995d67-9144-429c-b655-0c1fd37ad11c-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:49:24.226717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.226522    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdpqr\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-kube-api-access-zdpqr\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:49:24.226717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.226532    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04995d67-9144-429c-b655-0c1fd37ad11c-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:49:24.226717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.226541    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04995d67-9144-429c-b655-0c1fd37ad11c-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:49:24.437934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.437882    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"]
Nov 01 12:49:24.443097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.443060    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-h56vb"]
Nov 01 12:49:24.621121 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:24.621030    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" path="/var/lib/kubelet/pods/04995d67-9144-429c-b655-0c1fd37ad11c/volumes"
Nov 01 12:49:27.050588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:27.050549    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"]
Nov 01 12:49:27.094653 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:27.094607    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"]
Nov 01 12:49:27.099358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:27.099329    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0bdb7eb-fz7nj"]
Nov 01 12:49:27.103888 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:27.103852    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2791dc32-6jtdg"]
Nov 01 12:49:28.544942 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:28.544910    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": read tcp 10.132.0.2:37712->10.132.0.163:15021: read: connection reset by peer" start-of-body=
Nov 01 12:49:28.545515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:28.544983    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": read tcp 10.132.0.2:37712->10.132.0.163:15021: read: connection reset by peer"
Nov 01 12:49:28.618755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:28.618723    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d4279a-f0fe-4658-a98f-b8143c1d37df" path="/var/lib/kubelet/pods/97d4279a-f0fe-4658-a98f-b8143c1d37df/volumes"
Nov 01 12:49:28.619244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:28.619228    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be4b9e99-2af5-465d-a539-3e410025a08f" path="/var/lib/kubelet/pods/be4b9e99-2af5-465d-a539-3e410025a08f/volumes"
Nov 01 12:49:29.078753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:29.078717    5725 generic.go:356] "Generic (PLEG): container finished" podID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerID="04758c8923fdf57090c3dec06241d0b93ccd419606908b7f3be00ff8a16f4406" exitCode=0
Nov 01 12:49:29.078939 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:29.078791    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" event={"ID":"800fc942-dd8e-41df-8754-55ae9ab9cf46","Type":"ContainerDied","Data":"04758c8923fdf57090c3dec06241d0b93ccd419606908b7f3be00ff8a16f4406"}
Nov 01 12:49:29.079317 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:29.079296    5725 scope.go:122] "RemoveContainer" containerID="04758c8923fdf57090c3dec06241d0b93ccd419606908b7f3be00ff8a16f4406"
Nov 01 12:49:29.081008 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:29.080978    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:49:29.081103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:29.081045    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:49:29.081103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:29.081094    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:49:30.094594 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:30.094546    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" event={"ID":"800fc942-dd8e-41df-8754-55ae9ab9cf46","Type":"ContainerStarted","Data":"67e7401149b4f83af7d991d52908b9f27fb8a2b22296e39da4924380edcf423a"}
Nov 01 12:49:30.801741 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:30.801706    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:49:30.801929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:30.801877    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 12:49:30.801983 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:30.801930    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 12:49:31.032285 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:49:31.032212    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:49:31.063503 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:31.063427    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"]
Nov 01 12:49:31.096319 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:31.096288    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df28901e-4f5k7"]
Nov 01 12:49:31.801289 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:31.801247    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 12:49:31.801517 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:31.801314    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 12:49:32.059863 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:32.059723    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"]
Nov 01 12:49:32.103385 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:32.103345    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aea6b6a-8wnpb"]
Nov 01 12:49:32.622132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:32.622091    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235e00ca-7d40-49eb-bfde-2b15163e5ab1" path="/var/lib/kubelet/pods/235e00ca-7d40-49eb-bfde-2b15163e5ab1/volumes"
Nov 01 12:49:32.622667 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:32.622649    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7186b20-4ae3-4cdd-90de-09282817dd58" path="/var/lib/kubelet/pods/d7186b20-4ae3-4cdd-90de-09282817dd58/volumes"
Nov 01 12:49:32.802211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:32.802175    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 12:49:32.802395 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:32.802251    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 12:49:33.801657 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:33.801616    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 12:49:33.802170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:33.802139    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 12:49:34.802162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:34.802131    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 01 12:49:34.802563 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:34.802182    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 01 12:49:35.805877 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:35.805842    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:49:35.806394 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:35.806287    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:49:35.806965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:35.806925    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:49:36.046744 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:36.046711    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"]
Nov 01 12:49:36.051429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:36.051401    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e76cbf6-725c6"]
Nov 01 12:49:36.618363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:36.618325    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d" path="/var/lib/kubelet/pods/3b2f5a6d-f878-45aa-81c6-ac638a7e9c5d/volumes"
Nov 01 12:49:38.047893 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:38.047856    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"]
Nov 01 12:49:38.094516 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:38.094483    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248038df-nlnr4"]
Nov 01 12:49:38.618807 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:38.618766    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89633d72-74e0-437d-819b-97f5f4143306" path="/var/lib/kubelet/pods/89633d72-74e0-437d-819b-97f5f4143306/volumes"
Nov 01 12:49:39.490577 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:39.490537    5725 scope.go:122] "RemoveContainer" containerID="1b9ad4a8088bb7a7d4fc983945beabebfe66b1fd348ab5c8a71645355da0f38c"
Nov 01 12:49:39.501804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:39.501783    5725 scope.go:122] "RemoveContainer" containerID="48448f3d49fc07de8445637ea0882863bea6515d6cb3aa6206c9f1e0135270b9"
Nov 01 12:49:39.512175 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:39.512151    5725 scope.go:122] "RemoveContainer" containerID="643cc7b3c27d9f9f01f0b654793b625061245b34c5ea66fdba6c22c186bb322b"
Nov 01 12:49:41.048170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:41.048117    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"]
Nov 01 12:49:41.055418 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:41.055384    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfabce44-m75x7"]
Nov 01 12:49:42.619078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:42.619014    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0477fd2f-de1f-4860-86dd-4489b71fceaa" path="/var/lib/kubelet/pods/0477fd2f-de1f-4860-86dd-4489b71fceaa/volumes"
Nov 01 12:49:47.044109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:47.044072    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"]
Nov 01 12:49:47.091040 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:47.091001    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805927e-b4hpk"]
Nov 01 12:49:48.303902 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.303853    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"]
Nov 01 12:49:48.305408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.305378    5725 cpu_manager.go:414] "containerMap: removing container" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy"
Nov 01 12:49:48.305408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.305402    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy"
Nov 01 12:49:48.305753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.305736    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="04995d67-9144-429c-b655-0c1fd37ad11c" containerName="istio-proxy"
Nov 01 12:49:48.312332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.312305    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.565129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.557311    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-0cd09f-openshift-default-dockercfg-nnzzn\""
Nov 01 12:49:48.574602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.574564    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzsx\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-kube-api-access-gkzsx\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.574879 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.574855    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-socket\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.590074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.574991    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-credential-socket\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.590440 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.590409    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-podinfo\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.590584 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.590566    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-envoy\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.590844 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.590826    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-token\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.591188 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.591161    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-certs\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.591353 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.591335    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b7d530-db61-4ec9-9647-deb92b22a82f-istiod-ca-cert\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.591662 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.591641    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-data\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.706123 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.706085    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-token\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.706427 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.706404    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-certs\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.706608 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.706584    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b7d530-db61-4ec9-9647-deb92b22a82f-istiod-ca-cert\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.706888 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.706872    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-data\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.707119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.707102    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzsx\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-kube-api-access-gkzsx\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.707263 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.707246    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-socket\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.707385 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.707371    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-credential-socket\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.707581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.707562    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-podinfo\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.707709 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.707694    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-envoy\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.708947 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.708923    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-data\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.713171 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.709901    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-certs\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.713316 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.710626    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-socket\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.713408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.711149    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b7d530-db61-4ec9-9647-deb92b22a82f-istiod-ca-cert\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.721561 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.721174    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-credential-socket\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.721897 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.721875    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-envoy\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.724352 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.724274    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-podinfo\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.866166 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.866000    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzsx\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-kube-api-access-gkzsx\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.878173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.878026    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0266bd6f-3bc0-45e4-b6d4-efcb2220776a" path="/var/lib/kubelet/pods/0266bd6f-3bc0-45e4-b6d4-efcb2220776a/volumes"
Nov 01 12:49:48.879762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.879727    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"]
Nov 01 12:49:48.881976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.881913    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-token\") pod \"e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") " pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:48.945382 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:48.945106    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:49.488171 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:49.479718    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"]
Nov 01 12:49:49.511671 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:49.503386    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:49:49.511671 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:49.503498    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:49:49.511671 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:49.503545    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:49:49.764560 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:49.764528    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" event={"ID":"c0b7d530-db61-4ec9-9647-deb92b22a82f","Type":"ContainerStarted","Data":"2f3497d67ec79e2693608d11a48929dae654623db80706f26c583caf8aa7178a"}
Nov 01 12:49:50.778271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:50.778232    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" event={"ID":"c0b7d530-db61-4ec9-9647-deb92b22a82f","Type":"ContainerStarted","Data":"c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2"}
Nov 01 12:49:50.950073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:50.947396    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:50.972071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:50.961477    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:51.030727 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:51.025169    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" podStartSLOduration=3.025148974 podStartE2EDuration="3.025148974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:49:50.941014391 +0000 UTC m=+1632.448205613" watchObservedRunningTime="2026-11-01 12:49:51.025148974 +0000 UTC m=+1632.532340193"
Nov 01 12:49:51.787479 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:51.787446    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:51.788434 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:51.788415    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:49:52.044556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:52.044480    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"]
Nov 01 12:49:52.049655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:52.049627    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e754e361-r5cnw"]
Nov 01 12:49:52.617827 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:52.617790    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6867e3c2-5bde-456d-a5d7-70640987a35f" path="/var/lib/kubelet/pods/6867e3c2-5bde-456d-a5d7-70640987a35f/volumes"
Nov 01 12:49:58.156109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:58.156043    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"]
Nov 01 12:49:58.163681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:58.163644    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c752746-cvl9x"]
Nov 01 12:49:58.748821 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:49:58.748712    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c32452-c11e-4a58-ac55-e9841a6775b3" path="/var/lib/kubelet/pods/f9c32452-c11e-4a58-ac55-e9841a6775b3/volumes"
Nov 01 12:50:01.028890 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:01.028851    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:50:08.964735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:08.964527    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"]
Nov 01 12:50:08.999910 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:08.974824    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.054072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.035129    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.054414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.054382    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.054588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.054571    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.054700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.054687    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.054808 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.054794    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.054989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.054974    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.055289 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.055269    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.055447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.055430    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn82s\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-kube-api-access-mn82s\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.055584 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.055569    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49d-a997-42c7-9d2f-52c6dacdd881-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.142256 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.056007    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-187d06ed-openshift-default-dockercfg-h6ggd\""
Nov 01 12:50:09.246468 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.246410    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.246765 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.246743    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn82s\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-kube-api-access-mn82s\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.246918 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.246903    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49d-a997-42c7-9d2f-52c6dacdd881-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.247250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.247233    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.247404 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.247388    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.247558 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.247543    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.247678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.247648    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.247795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.247780    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.247958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.247944    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.248623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.248600    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.273300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.266699    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.273300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.267144    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"]
Nov 01 12:50:09.273300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.267449    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.273300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.267589    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.273300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.267838    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.273300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.268120    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49d-a997-42c7-9d2f-52c6dacdd881-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.273300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.273231    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.326213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.326134    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn82s\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-kube-api-access-mn82s\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.428494 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.428460    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:09.672134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:09.672017    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:10.150484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:10.149109    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"]
Nov 01 12:50:10.209465 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:10.209408    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:50:10.209769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:10.209744    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:50:10.209993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:10.209972    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:50:10.977859 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:10.977822    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" event={"ID":"385cf49d-a997-42c7-9d2f-52c6dacdd881","Type":"ContainerStarted","Data":"91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff"}
Nov 01 12:50:10.978038 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:10.977896    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" event={"ID":"385cf49d-a997-42c7-9d2f-52c6dacdd881","Type":"ContainerStarted","Data":"c957021583f7a38ec1e081f72813931ca2bbd602b69451527b9f73cb2f9d900e"}
Nov 01 12:50:11.090605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:11.090545    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" podStartSLOduration=3.090527099 podStartE2EDuration="3.090527099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:50:11.089265656 +0000 UTC m=+1652.596456897" watchObservedRunningTime="2026-11-01 12:50:11.090527099 +0000 UTC m=+1652.597718318"
Nov 01 12:50:11.673043 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:11.672999    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:11.679638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:11.679608    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:11.989729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:11.989661    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:11.990535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:11.990496    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:50:19.294024 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:19.293984    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"]
Nov 01 12:50:19.297831 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:19.297768    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy" containerID="cri-o://7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620" gracePeriod=30
Nov 01 12:50:19.349575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:19.346211    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"]
Nov 01 12:50:19.349575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:19.346467    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy" containerID="cri-o://ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc" gracePeriod=30
Nov 01 12:50:19.349575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:19.348311    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"]
Nov 01 12:50:19.349575 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:19.348511    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy" containerID="cri-o://67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f" gracePeriod=30
Nov 01 12:50:19.384326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:19.384287    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"]
Nov 01 12:50:19.384542 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:19.384515    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" containerID="cri-o://60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298" gracePeriod=30
Nov 01 12:50:21.425560 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:21.425520    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"]
Nov 01 12:50:21.426043 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:21.425764    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy" containerID="cri-o://2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc" gracePeriod=30
Nov 01 12:50:21.473513 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:21.473472    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"]
Nov 01 12:50:21.473720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:21.473682    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy" containerID="cri-o://0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae" gracePeriod=30
Nov 01 12:50:24.051870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.051822    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:50:24.052358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.051897    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:50:24.329508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.329425    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:50:24.329763 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.329728    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:50:24.775247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.773709    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"]
Nov 01 12:50:24.783606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.783014    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"]
Nov 01 12:50:24.783606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.783384    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:24.839614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.839353    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:50:24.839614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.839463    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:50:24.871790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.871745    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:24.871928 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.871795    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshps\" (UniqueName: \"kubernetes.io/projected/0b709ad8-e158-43ff-a23e-189629834ef2-kube-api-access-lshps\") pod \"maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:24.871928 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.871851    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:24.908721 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.908685    5725 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 12:50:24.908870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.908746    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 12:50:24.978308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.978271    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:24.978473 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.978324    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshps\" (UniqueName: \"kubernetes.io/projected/0b709ad8-e158-43ff-a23e-189629834ef2-kube-api-access-lshps\") pod \"maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:24.978473 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.978384    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:24.979462 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.979436    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:24.982016 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.981985    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:24.989346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:24.989320    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:50:25.033654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.033538    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshps\" (UniqueName: \"kubernetes.io/projected/0b709ad8-e158-43ff-a23e-189629834ef2-kube-api-access-lshps\") pod \"maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:25.081077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.080341    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-envoy\") pod \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") "
Nov 01 12:50:25.081077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.080410    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istiod-ca-cert\") pod \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") "
Nov 01 12:50:25.081077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.080583    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-token\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-token\") pod \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") "
Nov 01 12:50:25.081077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.080681    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-credential-socket\") pod \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") "
Nov 01 12:50:25.081077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.080748    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-data\") pod \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") "
Nov 01 12:50:25.081077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.080851    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-certs\") pod \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") "
Nov 01 12:50:25.081077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.080883    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-podinfo\") pod \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") "
Nov 01 12:50:25.081077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.080973    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-socket\") pod \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") "
Nov 01 12:50:25.081077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.081034    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-kube-api-access-6g5qs\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-kube-api-access-6g5qs\") pod \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\" (UID: \"b56ceeda-6a87-4ebb-b482-df5f1bae1706\") "
Nov 01 12:50:25.088101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.082546    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-certs" pod "b56ceeda-6a87-4ebb-b482-df5f1bae1706" (UID: "b56ceeda-6a87-4ebb-b482-df5f1bae1706"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.088101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.082775    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-data" pod "b56ceeda-6a87-4ebb-b482-df5f1bae1706" (UID: "b56ceeda-6a87-4ebb-b482-df5f1bae1706"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.088101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.082878    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istiod-ca-cert" pod "b56ceeda-6a87-4ebb-b482-df5f1bae1706" (UID: "b56ceeda-6a87-4ebb-b482-df5f1bae1706"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:25.088101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.083274    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-credential-socket" pod "b56ceeda-6a87-4ebb-b482-df5f1bae1706" (UID: "b56ceeda-6a87-4ebb-b482-df5f1bae1706"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.088101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.083413    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-socket" pod "b56ceeda-6a87-4ebb-b482-df5f1bae1706" (UID: "b56ceeda-6a87-4ebb-b482-df5f1bae1706"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.088875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.088852    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-podinfo" pod "b56ceeda-6a87-4ebb-b482-df5f1bae1706" (UID: "b56ceeda-6a87-4ebb-b482-df5f1bae1706"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:25.088984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.088881    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-kube-api-access-6g5qs" pod "b56ceeda-6a87-4ebb-b482-df5f1bae1706" (UID: "b56ceeda-6a87-4ebb-b482-df5f1bae1706"). InnerVolumeSpecName "kube-api-access-6g5qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:25.088984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.088908    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-token" pod "b56ceeda-6a87-4ebb-b482-df5f1bae1706" (UID: "b56ceeda-6a87-4ebb-b482-df5f1bae1706"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:25.090740 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.090690    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-envoy" pod "b56ceeda-6a87-4ebb-b482-df5f1bae1706" (UID: "b56ceeda-6a87-4ebb-b482-df5f1bae1706"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.112425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.112398    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:50:25.120571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.120542    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:50:25.122790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.122767    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:50:25.128663 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.128635    5725 generic.go:356] "Generic (PLEG): container finished" podID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerID="60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298" exitCode=0
Nov 01 12:50:25.128794 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.128696    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"
Nov 01 12:50:25.128870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.128805    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" event={"ID":"3bd0b429-5570-493d-8de7-9a315bae15c2","Type":"ContainerDied","Data":"60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298"}
Nov 01 12:50:25.128870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.128853    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48" event={"ID":"3bd0b429-5570-493d-8de7-9a315bae15c2","Type":"ContainerDied","Data":"aded86bd8e8cc80761208a82a9c3f8100fc8cd3eb4b82ef4e770f2e0a0e1e22c"}
Nov 01 12:50:25.128983 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.128884    5725 scope.go:122] "RemoveContainer" containerID="60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298"
Nov 01 12:50:25.130767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.130744    5725 generic.go:356] "Generic (PLEG): container finished" podID="64452fea-f72f-436f-b46d-ee9174d0f797" containerID="7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620" exitCode=0
Nov 01 12:50:25.130998 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.130914    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" event={"ID":"64452fea-f72f-436f-b46d-ee9174d0f797","Type":"ContainerDied","Data":"7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620"}
Nov 01 12:50:25.130998 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.130961    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7" event={"ID":"64452fea-f72f-436f-b46d-ee9174d0f797","Type":"ContainerDied","Data":"66dfaaf6b517a459adbd1aed4045206f65149f93cdb102a6cac705898fbccc9b"}
Nov 01 12:50:25.131318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.131038    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"
Nov 01 12:50:25.133889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.133323    5725 generic.go:356] "Generic (PLEG): container finished" podID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerID="ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc" exitCode=0
Nov 01 12:50:25.133889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.133405    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" event={"ID":"b56ceeda-6a87-4ebb-b482-df5f1bae1706","Type":"ContainerDied","Data":"ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc"}
Nov 01 12:50:25.133889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.133443    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8" event={"ID":"b56ceeda-6a87-4ebb-b482-df5f1bae1706","Type":"ContainerDied","Data":"3fabe9b7162bd17189714573dd9c0247050bd768502751e41983a73774ffc805"}
Nov 01 12:50:25.133889 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.133498    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"
Nov 01 12:50:25.141810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.141546    5725 generic.go:356] "Generic (PLEG): container finished" podID="488973be-421e-4a45-929c-6538515e36b3" containerID="67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f" exitCode=0
Nov 01 12:50:25.141810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.141673    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" event={"ID":"488973be-421e-4a45-929c-6538515e36b3","Type":"ContainerDied","Data":"67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f"}
Nov 01 12:50:25.141810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.141715    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn" event={"ID":"488973be-421e-4a45-929c-6538515e36b3","Type":"ContainerDied","Data":"1230264d9527dd7d364f2e6e5fab6e4181d347e536a7b064444614602cb56676"}
Nov 01 12:50:25.142264 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.142082    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"
Nov 01 12:50:25.155320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.155300    5725 scope.go:122] "RemoveContainer" containerID="60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298"
Nov 01 12:50:25.156472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.155880    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:25.156858 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:25.156836    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298\": container with ID starting with 60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298 not found: ID does not exist" containerID="60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298"
Nov 01 12:50:25.157037 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.157010    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298"} err="failed to get container status \"60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298\": rpc error: code = NotFound desc = could not find container \"60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298\": container with ID starting with 60ba81e6d005f64eccdb9a1556362fcdba8daa6858acd34885698951f3e8a298 not found: ID does not exist"
Nov 01 12:50:25.157205 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.157190    5725 scope.go:122] "RemoveContainer" containerID="7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620"
Nov 01 12:50:25.179999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.179972    5725 scope.go:122] "RemoveContainer" containerID="7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620"
Nov 01 12:50:25.180382 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:25.180360    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620\": container with ID starting with 7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620 not found: ID does not exist" containerID="7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620"
Nov 01 12:50:25.180477 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.180389    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620"} err="failed to get container status \"7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620\": rpc error: code = NotFound desc = could not find container \"7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620\": container with ID starting with 7adf387a19dfef30b8ba79e18104210c48fd1f31d16f2a64c3376d5bb48a3620 not found: ID does not exist"
Nov 01 12:50:25.180477 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.180406    5725 scope.go:122] "RemoveContainer" containerID="ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc"
Nov 01 12:50:25.183857 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.183827    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.183857 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.183860    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.184039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.183875    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.184039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.183891    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.184039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.183906    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.184039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.183920    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g5qs\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-kube-api-access-6g5qs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.184039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.183934    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.184039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.183948    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.184039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.183960    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56ceeda-6a87-4ebb-b482-df5f1bae1706-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.193995 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.193972    5725 scope.go:122] "RemoveContainer" containerID="ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc"
Nov 01 12:50:25.194446 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:25.194298    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc\": container with ID starting with ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc not found: ID does not exist" containerID="ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc"
Nov 01 12:50:25.194446 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.194341    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc"} err="failed to get container status \"ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc\": rpc error: code = NotFound desc = could not find container \"ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc\": container with ID starting with ab109dc9c6048f7d7592e9032b8f3ab9c856dce2fe3099a241edf2ffeec549fc not found: ID does not exist"
Nov 01 12:50:25.194446 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.194365    5725 scope.go:122] "RemoveContainer" containerID="67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f"
Nov 01 12:50:25.208330 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.208303    5725 scope.go:122] "RemoveContainer" containerID="67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f"
Nov 01 12:50:25.208837 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:25.208695    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f\": container with ID starting with 67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f not found: ID does not exist" containerID="67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f"
Nov 01 12:50:25.208837 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.208806    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f"} err="failed to get container status \"67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f\": rpc error: code = NotFound desc = could not find container \"67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f\": container with ID starting with 67d7429376c67c8e1a71437dc572354e99dcb516f7ea123990655dc2426e776f not found: ID does not exist"
Nov 01 12:50:25.286174 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.286139    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-kube-api-access-v64fb\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-kube-api-access-v64fb\") pod \"3bd0b429-5570-493d-8de7-9a315bae15c2\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") "
Nov 01 12:50:25.286815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.286791    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-data\") pod \"488973be-421e-4a45-929c-6538515e36b3\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") "
Nov 01 12:50:25.287366 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.287346    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-token\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-token\") pod \"3bd0b429-5570-493d-8de7-9a315bae15c2\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") "
Nov 01 12:50:25.287908 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.287290    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-data" pod "488973be-421e-4a45-929c-6538515e36b3" (UID: "488973be-421e-4a45-929c-6538515e36b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.288408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.288388    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/488973be-421e-4a45-929c-6538515e36b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488973be-421e-4a45-929c-6538515e36b3-istio-podinfo\") pod \"488973be-421e-4a45-929c-6538515e36b3\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") "
Nov 01 12:50:25.288586 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.288571    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-socket\") pod \"64452fea-f72f-436f-b46d-ee9174d0f797\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") "
Nov 01 12:50:25.289123 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.289104    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-data\") pod \"3bd0b429-5570-493d-8de7-9a315bae15c2\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") "
Nov 01 12:50:25.289504 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.289488    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-socket\") pod \"3bd0b429-5570-493d-8de7-9a315bae15c2\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") "
Nov 01 12:50:25.289877 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.289862    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64452fea-f72f-436f-b46d-ee9174d0f797-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64452fea-f72f-436f-b46d-ee9174d0f797-istiod-ca-cert\") pod \"64452fea-f72f-436f-b46d-ee9174d0f797\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") "
Nov 01 12:50:25.290408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.290390    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-kube-api-access-djg5h\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-kube-api-access-djg5h\") pod \"64452fea-f72f-436f-b46d-ee9174d0f797\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") "
Nov 01 12:50:25.290917 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.288972    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-socket" pod "64452fea-f72f-436f-b46d-ee9174d0f797" (UID: "64452fea-f72f-436f-b46d-ee9174d0f797"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.290917 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.289430    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-data" pod "3bd0b429-5570-493d-8de7-9a315bae15c2" (UID: "3bd0b429-5570-493d-8de7-9a315bae15c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.290917 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.289777    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-socket" pod "3bd0b429-5570-493d-8de7-9a315bae15c2" (UID: "3bd0b429-5570-493d-8de7-9a315bae15c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.290917 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.290335    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64452fea-f72f-436f-b46d-ee9174d0f797-istiod-ca-cert" pod "64452fea-f72f-436f-b46d-ee9174d0f797" (UID: "64452fea-f72f-436f-b46d-ee9174d0f797"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:25.291479 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.291462    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-certs\") pod \"64452fea-f72f-436f-b46d-ee9174d0f797\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") "
Nov 01 12:50:25.291931 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.291914    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-envoy\") pod \"3bd0b429-5570-493d-8de7-9a315bae15c2\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") "
Nov 01 12:50:25.292033 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.291969    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-istio-token\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-istio-token\") pod \"488973be-421e-4a45-929c-6538515e36b3\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") "
Nov 01 12:50:25.292155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292031    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-credential-socket\") pod \"488973be-421e-4a45-929c-6538515e36b3\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") "
Nov 01 12:50:25.292155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292079    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-credential-socket\") pod \"3bd0b429-5570-493d-8de7-9a315bae15c2\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") "
Nov 01 12:50:25.292155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292105    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-certs\") pod \"3bd0b429-5570-493d-8de7-9a315bae15c2\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") "
Nov 01 12:50:25.292312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292165    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd0b429-5570-493d-8de7-9a315bae15c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd0b429-5570-493d-8de7-9a315bae15c2-istiod-ca-cert\") pod \"3bd0b429-5570-493d-8de7-9a315bae15c2\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") "
Nov 01 12:50:25.292312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292196    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-istio-token\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-istio-token\") pod \"64452fea-f72f-436f-b46d-ee9174d0f797\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") "
Nov 01 12:50:25.292312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292240    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-envoy\") pod \"488973be-421e-4a45-929c-6538515e36b3\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") "
Nov 01 12:50:25.292312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292277    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-podinfo\") pod \"3bd0b429-5570-493d-8de7-9a315bae15c2\" (UID: \"3bd0b429-5570-493d-8de7-9a315bae15c2\") "
Nov 01 12:50:25.292526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292319    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-certs\") pod \"488973be-421e-4a45-929c-6538515e36b3\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") "
Nov 01 12:50:25.292526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292399    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488973be-421e-4a45-929c-6538515e36b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488973be-421e-4a45-929c-6538515e36b3-istiod-ca-cert\") pod \"488973be-421e-4a45-929c-6538515e36b3\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") "
Nov 01 12:50:25.292526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292452    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-envoy\") pod \"64452fea-f72f-436f-b46d-ee9174d0f797\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") "
Nov 01 12:50:25.292526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292499    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-data\") pod \"64452fea-f72f-436f-b46d-ee9174d0f797\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") "
Nov 01 12:50:25.292729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292576    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64452fea-f72f-436f-b46d-ee9174d0f797-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64452fea-f72f-436f-b46d-ee9174d0f797-istio-podinfo\") pod \"64452fea-f72f-436f-b46d-ee9174d0f797\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") "
Nov 01 12:50:25.292729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292628    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-kube-api-access-wxkm8\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-kube-api-access-wxkm8\") pod \"488973be-421e-4a45-929c-6538515e36b3\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") "
Nov 01 12:50:25.292729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292671    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-socket\") pod \"488973be-421e-4a45-929c-6538515e36b3\" (UID: \"488973be-421e-4a45-929c-6538515e36b3\") "
Nov 01 12:50:25.292729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.292709    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-credential-socket\") pod \"64452fea-f72f-436f-b46d-ee9174d0f797\" (UID: \"64452fea-f72f-436f-b46d-ee9174d0f797\") "
Nov 01 12:50:25.293488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.291857    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-certs" pod "64452fea-f72f-436f-b46d-ee9174d0f797" (UID: "64452fea-f72f-436f-b46d-ee9174d0f797"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.293574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.293460    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-credential-socket" pod "488973be-421e-4a45-929c-6538515e36b3" (UID: "488973be-421e-4a45-929c-6538515e36b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.294290 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.294268    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-data" pod "64452fea-f72f-436f-b46d-ee9174d0f797" (UID: "64452fea-f72f-436f-b46d-ee9174d0f797"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.296480 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.296291    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-certs" pod "3bd0b429-5570-493d-8de7-9a315bae15c2" (UID: "3bd0b429-5570-493d-8de7-9a315bae15c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.297432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.296967    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd0b429-5570-493d-8de7-9a315bae15c2-istiod-ca-cert" pod "3bd0b429-5570-493d-8de7-9a315bae15c2" (UID: "3bd0b429-5570-493d-8de7-9a315bae15c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:25.297817 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.297557    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-credential-socket" pod "3bd0b429-5570-493d-8de7-9a315bae15c2" (UID: "3bd0b429-5570-493d-8de7-9a315bae15c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.298231 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.297954    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-certs" pod "488973be-421e-4a45-929c-6538515e36b3" (UID: "488973be-421e-4a45-929c-6538515e36b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.302675 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.302469    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-kube-api-access-djg5h" pod "64452fea-f72f-436f-b46d-ee9174d0f797" (UID: "64452fea-f72f-436f-b46d-ee9174d0f797"). InnerVolumeSpecName "kube-api-access-djg5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:25.302675 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.302506    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488973be-421e-4a45-929c-6538515e36b3-istiod-ca-cert" pod "488973be-421e-4a45-929c-6538515e36b3" (UID: "488973be-421e-4a45-929c-6538515e36b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:25.302675 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.302591    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-envoy" pod "3bd0b429-5570-493d-8de7-9a315bae15c2" (UID: "3bd0b429-5570-493d-8de7-9a315bae15c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.302892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.302766    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-socket" pod "488973be-421e-4a45-929c-6538515e36b3" (UID: "488973be-421e-4a45-929c-6538515e36b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.302892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.302837    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-credential-socket" pod "64452fea-f72f-436f-b46d-ee9174d0f797" (UID: "64452fea-f72f-436f-b46d-ee9174d0f797"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303116    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-envoy" pod "488973be-421e-4a45-929c-6538515e36b3" (UID: "488973be-421e-4a45-929c-6538515e36b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303503    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488973be-421e-4a45-929c-6538515e36b3-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303524    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303538    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303555    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303570    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303585    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303599    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303613    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303627    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64452fea-f72f-436f-b46d-ee9174d0f797-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303641    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djg5h\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-kube-api-access-djg5h\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303656    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303670    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303684    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303697    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303710    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd0b429-5570-493d-8de7-9a315bae15c2-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303724    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd0b429-5570-493d-8de7-9a315bae15c2-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303737    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.303784 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.303759    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488973be-421e-4a45-929c-6538515e36b3-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.306925 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.306886    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-envoy" pod "64452fea-f72f-436f-b46d-ee9174d0f797" (UID: "64452fea-f72f-436f-b46d-ee9174d0f797"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:25.307214 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.307179    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-token" pod "3bd0b429-5570-493d-8de7-9a315bae15c2" (UID: "3bd0b429-5570-493d-8de7-9a315bae15c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:25.307470 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.307436    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-kube-api-access-v64fb" pod "3bd0b429-5570-493d-8de7-9a315bae15c2" (UID: "3bd0b429-5570-493d-8de7-9a315bae15c2"). InnerVolumeSpecName "kube-api-access-v64fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:25.312416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.312382    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-podinfo" pod "3bd0b429-5570-493d-8de7-9a315bae15c2" (UID: "3bd0b429-5570-493d-8de7-9a315bae15c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:25.312416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.312392    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-istio-token" pod "488973be-421e-4a45-929c-6538515e36b3" (UID: "488973be-421e-4a45-929c-6538515e36b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:25.312749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.312659    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/488973be-421e-4a45-929c-6538515e36b3-istio-podinfo" pod "488973be-421e-4a45-929c-6538515e36b3" (UID: "488973be-421e-4a45-929c-6538515e36b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:25.312749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.312680    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-istio-token" pod "64452fea-f72f-436f-b46d-ee9174d0f797" (UID: "64452fea-f72f-436f-b46d-ee9174d0f797"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:25.312749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.312720    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64452fea-f72f-436f-b46d-ee9174d0f797-istio-podinfo" pod "64452fea-f72f-436f-b46d-ee9174d0f797" (UID: "64452fea-f72f-436f-b46d-ee9174d0f797"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:25.314371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.314344    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-kube-api-access-wxkm8" pod "488973be-421e-4a45-929c-6538515e36b3" (UID: "488973be-421e-4a45-929c-6538515e36b3"). InnerVolumeSpecName "kube-api-access-wxkm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:25.398822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.398786    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"]
Nov 01 12:50:25.409649 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.409607    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-zpzp8"]
Nov 01 12:50:25.411237 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.411155    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v64fb\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-kube-api-access-v64fb\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.411438 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.411410    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.411438 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.411441    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488973be-421e-4a45-929c-6538515e36b3-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.411630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.411453    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.411630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.411466    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64452fea-f72f-436f-b46d-ee9174d0f797-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.411630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.411480    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd0b429-5570-493d-8de7-9a315bae15c2-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.411630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.411493    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64452fea-f72f-436f-b46d-ee9174d0f797-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.411630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.411507    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64452fea-f72f-436f-b46d-ee9174d0f797-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.411630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.411522    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxkm8\" (UniqueName: \"kubernetes.io/projected/488973be-421e-4a45-929c-6538515e36b3-kube-api-access-wxkm8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:25.612773 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.612691    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"]
Nov 01 12:50:25.619516 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:50:25.619488    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b709ad8_e158_43ff_a23e_189629834ef2.slice/crio-35d5b7d3bac2d96421a8331ed3d52ffa365d1f057a797f555c92669baf063eb9 WatchSource:0}: Error finding container 35d5b7d3bac2d96421a8331ed3d52ffa365d1f057a797f555c92669baf063eb9: Status 404 returned error can't find the container with id 35d5b7d3bac2d96421a8331ed3d52ffa365d1f057a797f555c92669baf063eb9
Nov 01 12:50:25.768183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.768112    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"]
Nov 01 12:50:25.782112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.779964    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-nskl7"]
Nov 01 12:50:25.874640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.874558    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"]
Nov 01 12:50:25.879401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.879368    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-8bqtn"]
Nov 01 12:50:25.981044 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.981006    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"]
Nov 01 12:50:25.985259 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:25.985229    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-l8m48"]
Nov 01 12:50:26.153730 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.153628    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br" event={"ID":"0b709ad8-e158-43ff-a23e-189629834ef2","Type":"ContainerStarted","Data":"6967e82f7571d7cb9a88702398e51f971b2e2f173e8207ea4baf0a846a6a00f1"}
Nov 01 12:50:26.153730 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.153700    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br" event={"ID":"0b709ad8-e158-43ff-a23e-189629834ef2","Type":"ContainerStarted","Data":"35d5b7d3bac2d96421a8331ed3d52ffa365d1f057a797f555c92669baf063eb9"}
Nov 01 12:50:26.262557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.262502    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br" podStartSLOduration=2.262488045 podStartE2EDuration="2.262488045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:50:26.260231246 +0000 UTC m=+1667.767422464" watchObservedRunningTime="2026-11-01 12:50:26.262488045 +0000 UTC m=+1667.769679262"
Nov 01 12:50:26.621303 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.621271    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" path="/var/lib/kubelet/pods/3bd0b429-5570-493d-8de7-9a315bae15c2/volumes"
Nov 01 12:50:26.622096 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.622041    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488973be-421e-4a45-929c-6538515e36b3" path="/var/lib/kubelet/pods/488973be-421e-4a45-929c-6538515e36b3/volumes"
Nov 01 12:50:26.622683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.622668    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" path="/var/lib/kubelet/pods/64452fea-f72f-436f-b46d-ee9174d0f797/volumes"
Nov 01 12:50:26.623573 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.623555    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" path="/var/lib/kubelet/pods/b56ceeda-6a87-4ebb-b482-df5f1bae1706/volumes"
Nov 01 12:50:26.773468 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.773438    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:50:26.823661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.823632    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:50:26.935536 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935500    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-credential-socket\") pod \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") "
Nov 01 12:50:26.935706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935563    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-socket\") pod \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") "
Nov 01 12:50:26.935706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935664    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-credential-socket\") pod \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") "
Nov 01 12:50:26.935833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935736    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-certs\") pod \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") "
Nov 01 12:50:26.935833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935807    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-token\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-token\") pod \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") "
Nov 01 12:50:26.935951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935827    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-credential-socket" pod "a9dea2be-d11e-4622-8bcf-b9da24ce773d" (UID: "a9dea2be-d11e-4622-8bcf-b9da24ce773d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:26.935951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935843    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-socket" pod "a9dea2be-d11e-4622-8bcf-b9da24ce773d" (UID: "a9dea2be-d11e-4622-8bcf-b9da24ce773d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:26.935951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935840    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-kube-api-access-8c5fp\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-kube-api-access-8c5fp\") pod \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") "
Nov 01 12:50:26.935951 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935937    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-envoy\") pod \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") "
Nov 01 12:50:26.936360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.935999    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-certs\") pod \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") "
Nov 01 12:50:26.936360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.936039    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-envoy\") pod \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") "
Nov 01 12:50:26.936360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.936165    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-data\") pod \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") "
Nov 01 12:50:26.936360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.936197    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-podinfo\") pod \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") "
Nov 01 12:50:26.936360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.936245    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-token\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-token\") pod \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") "
Nov 01 12:50:26.937706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.936478    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-certs" pod "a9dea2be-d11e-4622-8bcf-b9da24ce773d" (UID: "a9dea2be-d11e-4622-8bcf-b9da24ce773d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:26.937706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.936617    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-certs" pod "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" (UID: "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:26.937706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.936699    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-data" pod "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" (UID: "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:26.937706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.937098    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-credential-socket" pod "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" (UID: "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:26.937706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.937577    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-kube-api-access-rg5nl\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-kube-api-access-rg5nl\") pod \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") "
Nov 01 12:50:26.937986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.937894    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istiod-ca-cert\") pod \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") "
Nov 01 12:50:26.937986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.937946    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-data\") pod \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\" (UID: \"a9dea2be-d11e-4622-8bcf-b9da24ce773d\") "
Nov 01 12:50:26.938122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.937996    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-podinfo\") pod \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") "
Nov 01 12:50:26.938122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.938030    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istiod-ca-cert\") pod \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") "
Nov 01 12:50:26.938122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.938078    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-socket\") pod \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\" (UID: \"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d\") "
Nov 01 12:50:26.938780 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.938760    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:26.938875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.938783    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:26.938875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.938797    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:26.938875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.938811    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:26.938875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.938824    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:26.938875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.938836    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:26.939244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.939073    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-socket" pod "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" (UID: "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:26.939244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.939226    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-data" pod "a9dea2be-d11e-4622-8bcf-b9da24ce773d" (UID: "a9dea2be-d11e-4622-8bcf-b9da24ce773d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:26.940069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.939521    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istiod-ca-cert" pod "a9dea2be-d11e-4622-8bcf-b9da24ce773d" (UID: "a9dea2be-d11e-4622-8bcf-b9da24ce773d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:26.940069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.939927    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istiod-ca-cert" pod "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" (UID: "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:26.940069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.939992    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-kube-api-access-8c5fp" pod "a9dea2be-d11e-4622-8bcf-b9da24ce773d" (UID: "a9dea2be-d11e-4622-8bcf-b9da24ce773d"). InnerVolumeSpecName "kube-api-access-8c5fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:26.940069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.940019    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-token" pod "a9dea2be-d11e-4622-8bcf-b9da24ce773d" (UID: "a9dea2be-d11e-4622-8bcf-b9da24ce773d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:26.940359 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.940308    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-token" pod "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" (UID: "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:26.941308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.941272    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-podinfo" pod "a9dea2be-d11e-4622-8bcf-b9da24ce773d" (UID: "a9dea2be-d11e-4622-8bcf-b9da24ce773d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:26.941531 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.941507    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-kube-api-access-rg5nl" pod "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" (UID: "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d"). InnerVolumeSpecName "kube-api-access-rg5nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:26.941726 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.941703    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-podinfo" pod "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" (UID: "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:26.942318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.942292    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-envoy" pod "a9dea2be-d11e-4622-8bcf-b9da24ce773d" (UID: "a9dea2be-d11e-4622-8bcf-b9da24ce773d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:26.942404 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:26.942377    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-envoy" pod "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" (UID: "b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040691    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040727    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5fp\" (UniqueName: \"kubernetes.io/projected/a9dea2be-d11e-4622-8bcf-b9da24ce773d-kube-api-access-8c5fp\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040739    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040747    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040755    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040765    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040773    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg5nl\" (UniqueName: \"kubernetes.io/projected/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-kube-api-access-rg5nl\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040781    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040790    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9dea2be-d11e-4622-8bcf-b9da24ce773d-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040798    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040807    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.040883 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.040815    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:27.048196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.048162    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"]
Nov 01 12:50:27.055229 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.055194    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c24e8e9b-624t2"]
Nov 01 12:50:27.169194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.169159    5725 generic.go:356] "Generic (PLEG): container finished" podID="0b709ad8-e158-43ff-a23e-189629834ef2" containerID="6967e82f7571d7cb9a88702398e51f971b2e2f173e8207ea4baf0a846a6a00f1" exitCode=0
Nov 01 12:50:27.169635 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.169244    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br" event={"ID":"0b709ad8-e158-43ff-a23e-189629834ef2","Type":"ContainerDied","Data":"6967e82f7571d7cb9a88702398e51f971b2e2f173e8207ea4baf0a846a6a00f1"}
Nov 01 12:50:27.170828 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.170806    5725 generic.go:356] "Generic (PLEG): container finished" podID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerID="2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc" exitCode=0
Nov 01 12:50:27.170953 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.170883    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"
Nov 01 12:50:27.170953 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.170918    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" event={"ID":"a9dea2be-d11e-4622-8bcf-b9da24ce773d","Type":"ContainerDied","Data":"2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc"}
Nov 01 12:50:27.171059 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.170969    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms" event={"ID":"a9dea2be-d11e-4622-8bcf-b9da24ce773d","Type":"ContainerDied","Data":"15719f821c260bf18941e1063121db14302a75bd1b3d05c021d419009c466fc8"}
Nov 01 12:50:27.171059 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.170998    5725 scope.go:122] "RemoveContainer" containerID="2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc"
Nov 01 12:50:27.172479 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.172457    5725 generic.go:356] "Generic (PLEG): container finished" podID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerID="0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae" exitCode=0
Nov 01 12:50:27.172582 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.172507    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"
Nov 01 12:50:27.172653 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.172577    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" event={"ID":"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d","Type":"ContainerDied","Data":"0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae"}
Nov 01 12:50:27.172696 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.172682    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf" event={"ID":"b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d","Type":"ContainerDied","Data":"e995aa8a5d844a240a6ba367e7b634516ac6929ea67c17ec162cb8e4b7c8976a"}
Nov 01 12:50:27.185919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.185898    5725 scope.go:122] "RemoveContainer" containerID="2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc"
Nov 01 12:50:27.186273 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:27.186253    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc\": container with ID starting with 2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc not found: ID does not exist" containerID="2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc"
Nov 01 12:50:27.186366 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.186286    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc"} err="failed to get container status \"2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc\": rpc error: code = NotFound desc = could not find container \"2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc\": container with ID starting with 2fc07448086fa7f970a5213c6dd8d4e658db8b99b171c0bec1779271f0cad8dc not found: ID does not exist"
Nov 01 12:50:27.186366 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.186313    5725 scope.go:122] "RemoveContainer" containerID="0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae"
Nov 01 12:50:27.196825 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.196807    5725 scope.go:122] "RemoveContainer" containerID="0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae"
Nov 01 12:50:27.197119 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:27.197096    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae\": container with ID starting with 0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae not found: ID does not exist" containerID="0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae"
Nov 01 12:50:27.197191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.197130    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae"} err="failed to get container status \"0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae\": rpc error: code = NotFound desc = could not find container \"0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae\": container with ID starting with 0b2741609b80641c35a936da7ba085c5c25e5b02cd732f2e056607283b258eae not found: ID does not exist"
Nov 01 12:50:27.385123 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.385014    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"]
Nov 01 12:50:27.390223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.390196    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cpfmf"]
Nov 01 12:50:27.494499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.494459    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"]
Nov 01 12:50:27.499589 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:27.499555    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-9rwms"]
Nov 01 12:50:28.357502 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.357478    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:28.463140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.463032    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-xks-ca\") pod \"0b709ad8-e158-43ff-a23e-189629834ef2\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") "
Nov 01 12:50:28.463140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.463128    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-openshift-ca\") pod \"0b709ad8-e158-43ff-a23e-189629834ef2\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") "
Nov 01 12:50:28.463371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.463292    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b709ad8-e158-43ff-a23e-189629834ef2-kube-api-access-lshps\" (UniqueName: \"kubernetes.io/projected/0b709ad8-e158-43ff-a23e-189629834ef2-kube-api-access-lshps\") pod \"0b709ad8-e158-43ff-a23e-189629834ef2\" (UID: \"0b709ad8-e158-43ff-a23e-189629834ef2\") "
Nov 01 12:50:28.463603 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.463570    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-openshift-ca" pod "0b709ad8-e158-43ff-a23e-189629834ef2" (UID: "0b709ad8-e158-43ff-a23e-189629834ef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:28.465667 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.465642    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:28.465788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.465668    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-xks-ca" pod "0b709ad8-e158-43ff-a23e-189629834ef2" (UID: "0b709ad8-e158-43ff-a23e-189629834ef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:50:28.466257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.466221    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b709ad8-e158-43ff-a23e-189629834ef2-kube-api-access-lshps" pod "0b709ad8-e158-43ff-a23e-189629834ef2" (UID: "0b709ad8-e158-43ff-a23e-189629834ef2"). InnerVolumeSpecName "kube-api-access-lshps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:28.567501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.567249    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b709ad8-e158-43ff-a23e-189629834ef2-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:28.567501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.567448    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lshps\" (UniqueName: \"kubernetes.io/projected/0b709ad8-e158-43ff-a23e-189629834ef2-kube-api-access-lshps\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:28.623312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.623258    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44747563-edf3-414b-8e28-e027b06dd779" path="/var/lib/kubelet/pods/44747563-edf3-414b-8e28-e027b06dd779/volumes"
Nov 01 12:50:28.623950 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.623924    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" path="/var/lib/kubelet/pods/a9dea2be-d11e-4622-8bcf-b9da24ce773d/volumes"
Nov 01 12:50:28.624445 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:28.624427    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" path="/var/lib/kubelet/pods/b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d/volumes"
Nov 01 12:50:29.146168 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.146130    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"]
Nov 01 12:50:29.147500 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147468    5725 cpu_manager.go:414] "containerMap: removing container" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy"
Nov 01 12:50:29.147500 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147498    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147517    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147525    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147562    5725 cpu_manager.go:414] "containerMap: removing container" podUID="0b709ad8-e158-43ff-a23e-189629834ef2" containerName="revoke-keys"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147569    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b709ad8-e158-43ff-a23e-189629834ef2" containerName="revoke-keys"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147590    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147597    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147629    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147636    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147650    5725 cpu_manager.go:414] "containerMap: removing container" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147658    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147671    5725 cpu_manager.go:414] "containerMap: removing container" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy"
Nov 01 12:50:29.147698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147678    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy"
Nov 01 12:50:29.148333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147834    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="488973be-421e-4a45-929c-6538515e36b3" containerName="istio-proxy"
Nov 01 12:50:29.148333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147850    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93cbe0a-0e18-4aae-b2dc-aeae691aeb3d" containerName="istio-proxy"
Nov 01 12:50:29.148333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147863    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b709ad8-e158-43ff-a23e-189629834ef2" containerName="revoke-keys"
Nov 01 12:50:29.148333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147874    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56ceeda-6a87-4ebb-b482-df5f1bae1706" containerName="istio-proxy"
Nov 01 12:50:29.148333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147886    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd0b429-5570-493d-8de7-9a315bae15c2" containerName="istio-proxy"
Nov 01 12:50:29.148333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147896    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9dea2be-d11e-4622-8bcf-b9da24ce773d" containerName="istio-proxy"
Nov 01 12:50:29.148333 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.147904    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="64452fea-f72f-436f-b46d-ee9174d0f797" containerName="istio-proxy"
Nov 01 12:50:29.153518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.153495    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.189776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.189738    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.189964 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.189885    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.190045 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.190025    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qnhb\" (UniqueName: \"kubernetes.io/projected/549b3ed6-875e-4d00-b16a-c5a4feb2736d-kube-api-access-8qnhb\") pod \"maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.197355 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.197317    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"
Nov 01 12:50:29.197355 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.197332    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br" event={"ID":"0b709ad8-e158-43ff-a23e-189629834ef2","Type":"ContainerDied","Data":"35d5b7d3bac2d96421a8331ed3d52ffa365d1f057a797f555c92669baf063eb9"}
Nov 01 12:50:29.197618 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.197389    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35d5b7d3bac2d96421a8331ed3d52ffa365d1f057a797f555c92669baf063eb9"
Nov 01 12:50:29.203253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.203227    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"]
Nov 01 12:50:29.291613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.291576    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.291826 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.291694    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qnhb\" (UniqueName: \"kubernetes.io/projected/549b3ed6-875e-4d00-b16a-c5a4feb2736d-kube-api-access-8qnhb\") pod \"maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.291978 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.291941    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.292826 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.292763    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.294360 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.294333    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.345630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.345588    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qnhb\" (UniqueName: \"kubernetes.io/projected/549b3ed6-875e-4d00-b16a-c5a4feb2736d-kube-api-access-8qnhb\") pod \"maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.470835 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.470801    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:29.850943 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:29.850857    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"]
Nov 01 12:50:29.856911 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:50:29.856878    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549b3ed6_875e_4d00_b16a_c5a4feb2736d.slice/crio-1a16e5e8cbe31f780d3fc0b68725d5db98f6aef1f28770af361badd2ed11d733 WatchSource:0}: Error finding container 1a16e5e8cbe31f780d3fc0b68725d5db98f6aef1f28770af361badd2ed11d733: Status 404 returned error can't find the container with id 1a16e5e8cbe31f780d3fc0b68725d5db98f6aef1f28770af361badd2ed11d733
Nov 01 12:50:30.207892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:30.207809    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc" event={"ID":"549b3ed6-875e-4d00-b16a-c5a4feb2736d","Type":"ContainerStarted","Data":"1a16e5e8cbe31f780d3fc0b68725d5db98f6aef1f28770af361badd2ed11d733"}
Nov 01 12:50:31.044162 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:31.044121    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:50:31.217923 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:31.217833    5725 generic.go:356] "Generic (PLEG): container finished" podID="549b3ed6-875e-4d00-b16a-c5a4feb2736d" containerID="f247258b818517d5f4b3875e436b4d5aecb0fc20bac16626af8cd72601c7340b" exitCode=0
Nov 01 12:50:31.218108 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:31.217916    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc" event={"ID":"549b3ed6-875e-4d00-b16a-c5a4feb2736d","Type":"ContainerDied","Data":"f247258b818517d5f4b3875e436b4d5aecb0fc20bac16626af8cd72601c7340b"}
Nov 01 12:50:32.402256 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.402226    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:32.544528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.543149    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-openshift-ca\") pod \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") "
Nov 01 12:50:32.544528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.543294    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549b3ed6-875e-4d00-b16a-c5a4feb2736d-kube-api-access-8qnhb\" (UniqueName: \"kubernetes.io/projected/549b3ed6-875e-4d00-b16a-c5a4feb2736d-kube-api-access-8qnhb\") pod \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") "
Nov 01 12:50:32.544528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.543346    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-xks-ca\") pod \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\" (UID: \"549b3ed6-875e-4d00-b16a-c5a4feb2736d\") "
Nov 01 12:50:32.544528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.543952    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-openshift-ca" pod "549b3ed6-875e-4d00-b16a-c5a4feb2736d" (UID: "549b3ed6-875e-4d00-b16a-c5a4feb2736d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:32.546606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.546570    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549b3ed6-875e-4d00-b16a-c5a4feb2736d-kube-api-access-8qnhb" pod "549b3ed6-875e-4d00-b16a-c5a4feb2736d" (UID: "549b3ed6-875e-4d00-b16a-c5a4feb2736d"). InnerVolumeSpecName "kube-api-access-8qnhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:32.548841 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.548803    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-xks-ca" pod "549b3ed6-875e-4d00-b16a-c5a4feb2736d" (UID: "549b3ed6-875e-4d00-b16a-c5a4feb2736d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:50:32.645772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.645727    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:32.645772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.645763    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qnhb\" (UniqueName: \"kubernetes.io/projected/549b3ed6-875e-4d00-b16a-c5a4feb2736d-kube-api-access-8qnhb\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:32.645772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:32.645777    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549b3ed6-875e-4d00-b16a-c5a4feb2736d-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:33.243595 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:33.243566    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"
Nov 01 12:50:33.243916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:33.243599    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc" event={"ID":"549b3ed6-875e-4d00-b16a-c5a4feb2736d","Type":"ContainerDied","Data":"1a16e5e8cbe31f780d3fc0b68725d5db98f6aef1f28770af361badd2ed11d733"}
Nov 01 12:50:33.244072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:33.243947    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a16e5e8cbe31f780d3fc0b68725d5db98f6aef1f28770af361badd2ed11d733"
Nov 01 12:50:39.611073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.610568    5725 scope.go:122] "RemoveContainer" containerID="921c8342be234870c6dc811775ed8a05f8ef8eb6bd4e220e8e8ba38e2d7db8a1"
Nov 01 12:50:39.660602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.660578    5725 scope.go:122] "RemoveContainer" containerID="a03deeb01a14c746c96356ab29a5d5a7056655e8365c533f6234a067d3ff178a"
Nov 01 12:50:39.732468 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.732443    5725 scope.go:122] "RemoveContainer" containerID="5ebaf3abc1c1f7016c70c32be4c0f1ca76eaff779b6d0bf6f601e78e52be217a"
Nov 01 12:50:39.765243 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.765208    5725 scope.go:122] "RemoveContainer" containerID="9d8a9110b82934041f67af76e4eae22af9655465896a1e9b8f47410e191a38da"
Nov 01 12:50:39.778309 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.778289    5725 scope.go:122] "RemoveContainer" containerID="201730e3268c927734d7e5a12eb9aff23f3eb478a817a50e3de287ba437ce5b3"
Nov 01 12:50:39.788789 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.788766    5725 scope.go:122] "RemoveContainer" containerID="eeba5c19e18484db9c472070723f094df2224a9368092fd94705adb080e148ca"
Nov 01 12:50:39.799404 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.799385    5725 scope.go:122] "RemoveContainer" containerID="fdbed3f8f1140ab3cef12a88633821c657f0332f87731701e1fbd28bb41fd698"
Nov 01 12:50:39.809858 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.809834    5725 scope.go:122] "RemoveContainer" containerID="45115db1c3c41c22d9480e0b6d99dc0fc454191d0d3a4792357d2b4106efa2df"
Nov 01 12:50:39.823289 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.823263    5725 scope.go:122] "RemoveContainer" containerID="f90738678ffcd69b05fffcd82b544c160a206b6d0e27bec4a2f93a47d29f4ae4"
Nov 01 12:50:39.833633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.833611    5725 scope.go:122] "RemoveContainer" containerID="397b53b96b2b4edf75291bf30cf5a67666fe0ef71e0a070040635e5fcd6eb76a"
Nov 01 12:50:39.845966 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.845942    5725 scope.go:122] "RemoveContainer" containerID="f13466baca856d0616be8c9760e02bba87c2d8fad8184b18491b005b765be936"
Nov 01 12:50:39.856301 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.856279    5725 scope.go:122] "RemoveContainer" containerID="810e5b051ffd156805f3999421d5ffb95a33c1e23f5a17993fb1f7b71d2ea777"
Nov 01 12:50:39.866284 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.866263    5725 scope.go:122] "RemoveContainer" containerID="90015c5a40ce9aaf33c8c9cf0ada4b364f85519d6c33aa409a4e44c7ec7aad36"
Nov 01 12:50:39.879924 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.879898    5725 scope.go:122] "RemoveContainer" containerID="6dccd38bfa09c81883f4356026f298d766fa18b0db8ad37dbf589126e3d45f6b"
Nov 01 12:50:39.890153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:39.890127    5725 scope.go:122] "RemoveContainer" containerID="aab1105e2f77195093ccb0810bde4e689d031b889a64c74870351bdb3be47eca"
Nov 01 12:50:40.045652 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:40.045627    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"]
Nov 01 12:50:40.093444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:40.093413    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0430f-8n87d"]
Nov 01 12:50:40.622908 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:40.622872    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32afab9-a145-48f6-8559-d612b675f08d" path="/var/lib/kubelet/pods/f32afab9-a145-48f6-8559-d612b675f08d/volumes"
Nov 01 12:50:41.045753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:41.045715    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"]
Nov 01 12:50:41.092070 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:41.092024    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ed9b47-s9vz6"]
Nov 01 12:50:42.625429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:42.623619    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c057128e-776c-4691-9904-597c45635c81" path="/var/lib/kubelet/pods/c057128e-776c-4691-9904-597c45635c81/volumes"
Nov 01 12:50:49.452940 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:49.452910    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"]
Nov 01 12:50:49.453438 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:49.453115    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy" containerID="cri-o://85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b" gracePeriod=30
Nov 01 12:50:49.500672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:49.500637    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"]
Nov 01 12:50:49.500899 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:49.500873    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" containerID="cri-o://3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39" gracePeriod=30
Nov 01 12:50:49.505066 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:49.505021    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"]
Nov 01 12:50:49.505255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:49.505234    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy" containerID="cri-o://1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56" gracePeriod=30
Nov 01 12:50:51.031804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:51.031769    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:50:51.032327 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:51.031822    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:50:51.079086 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:51.079028    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:50:51.079258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:51.079094    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:50:54.372308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.372262    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:50:54.372808 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.372349    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:50:54.819201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.818788    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:50:54.887032 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.886996    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:50:54.913094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.913034    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:50:54.932590 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.932555    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-data\") pod \"417ed549-3aaa-47e6-b135-fa4317813918\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") "
Nov 01 12:50:54.932751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.932625    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417ed549-3aaa-47e6-b135-fa4317813918-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417ed549-3aaa-47e6-b135-fa4317813918-istiod-ca-cert\") pod \"417ed549-3aaa-47e6-b135-fa4317813918\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") "
Nov 01 12:50:54.932751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.932669    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/417ed549-3aaa-47e6-b135-fa4317813918-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417ed549-3aaa-47e6-b135-fa4317813918-istio-podinfo\") pod \"417ed549-3aaa-47e6-b135-fa4317813918\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") "
Nov 01 12:50:54.932882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.932764    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-certs\") pod \"417ed549-3aaa-47e6-b135-fa4317813918\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") "
Nov 01 12:50:54.932882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.932792    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-istio-token\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-istio-token\") pod \"417ed549-3aaa-47e6-b135-fa4317813918\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") "
Nov 01 12:50:54.932995 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.932883    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-envoy\") pod \"417ed549-3aaa-47e6-b135-fa4317813918\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") "
Nov 01 12:50:54.932995 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.932891    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-data" pod "417ed549-3aaa-47e6-b135-fa4317813918" (UID: "417ed549-3aaa-47e6-b135-fa4317813918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:54.933133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933005    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-kube-api-access-gdsx8\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-kube-api-access-gdsx8\") pod \"417ed549-3aaa-47e6-b135-fa4317813918\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") "
Nov 01 12:50:54.933133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933109    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-credential-socket\") pod \"417ed549-3aaa-47e6-b135-fa4317813918\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") "
Nov 01 12:50:54.933253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933143    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417ed549-3aaa-47e6-b135-fa4317813918-istiod-ca-cert" pod "417ed549-3aaa-47e6-b135-fa4317813918" (UID: "417ed549-3aaa-47e6-b135-fa4317813918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:54.933253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933153    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-socket\") pod \"417ed549-3aaa-47e6-b135-fa4317813918\" (UID: \"417ed549-3aaa-47e6-b135-fa4317813918\") "
Nov 01 12:50:54.934000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933626    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-certs" pod "417ed549-3aaa-47e6-b135-fa4317813918" (UID: "417ed549-3aaa-47e6-b135-fa4317813918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:54.934000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933862    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:54.934000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933886    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:54.934000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933901    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417ed549-3aaa-47e6-b135-fa4317813918-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:54.934000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933967    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-credential-socket" pod "417ed549-3aaa-47e6-b135-fa4317813918" (UID: "417ed549-3aaa-47e6-b135-fa4317813918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:54.934000 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.933990    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-socket" pod "417ed549-3aaa-47e6-b135-fa4317813918" (UID: "417ed549-3aaa-47e6-b135-fa4317813918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:54.935430 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.935407    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-kube-api-access-gdsx8" pod "417ed549-3aaa-47e6-b135-fa4317813918" (UID: "417ed549-3aaa-47e6-b135-fa4317813918"). InnerVolumeSpecName "kube-api-access-gdsx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:54.935649 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.935623    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-envoy" pod "417ed549-3aaa-47e6-b135-fa4317813918" (UID: "417ed549-3aaa-47e6-b135-fa4317813918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:54.935720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.935663    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/417ed549-3aaa-47e6-b135-fa4317813918-istio-podinfo" pod "417ed549-3aaa-47e6-b135-fa4317813918" (UID: "417ed549-3aaa-47e6-b135-fa4317813918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:54.935720 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:54.935673    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-istio-token" pod "417ed549-3aaa-47e6-b135-fa4317813918" (UID: "417ed549-3aaa-47e6-b135-fa4317813918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:55.039122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.038672    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-envoy\") pod \"22b06a90-c616-4913-9a07-52eec95b9826\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") "
Nov 01 12:50:55.039122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.038724    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b06a90-c616-4913-9a07-52eec95b9826-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b06a90-c616-4913-9a07-52eec95b9826-istiod-ca-cert\") pod \"22b06a90-c616-4913-9a07-52eec95b9826\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") "
Nov 01 12:50:55.039122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.038803    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-certs\") pod \"7d541627-125c-4a48-b599-129c78b75c94\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") "
Nov 01 12:50:55.039122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.038851    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-envoy\") pod \"7d541627-125c-4a48-b599-129c78b75c94\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") "
Nov 01 12:50:55.039122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.038882    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-socket\") pod \"7d541627-125c-4a48-b599-129c78b75c94\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") "
Nov 01 12:50:55.039122 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.038951    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-certs\") pod \"22b06a90-c616-4913-9a07-52eec95b9826\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") "
Nov 01 12:50:55.041775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.039471    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-certs" pod "7d541627-125c-4a48-b599-129c78b75c94" (UID: "7d541627-125c-4a48-b599-129c78b75c94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.041775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.039670    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-certs" pod "22b06a90-c616-4913-9a07-52eec95b9826" (UID: "22b06a90-c616-4913-9a07-52eec95b9826"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.041775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.039728    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-socket" pod "7d541627-125c-4a48-b599-129c78b75c94" (UID: "7d541627-125c-4a48-b599-129c78b75c94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.041775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.040146    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b06a90-c616-4913-9a07-52eec95b9826-istiod-ca-cert" pod "22b06a90-c616-4913-9a07-52eec95b9826" (UID: "22b06a90-c616-4913-9a07-52eec95b9826"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.041849    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-data\") pod \"7d541627-125c-4a48-b599-129c78b75c94\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.041889    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22b06a90-c616-4913-9a07-52eec95b9826-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b06a90-c616-4913-9a07-52eec95b9826-istio-podinfo\") pod \"22b06a90-c616-4913-9a07-52eec95b9826\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.041936    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-socket\") pod \"22b06a90-c616-4913-9a07-52eec95b9826\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.041976    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d541627-125c-4a48-b599-129c78b75c94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d541627-125c-4a48-b599-129c78b75c94-istio-podinfo\") pod \"7d541627-125c-4a48-b599-129c78b75c94\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042000    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d541627-125c-4a48-b599-129c78b75c94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d541627-125c-4a48-b599-129c78b75c94-istiod-ca-cert\") pod \"7d541627-125c-4a48-b599-129c78b75c94\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042075    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-kube-api-access-d9zld\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-kube-api-access-d9zld\") pod \"22b06a90-c616-4913-9a07-52eec95b9826\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042113    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-istio-token\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-istio-token\") pod \"22b06a90-c616-4913-9a07-52eec95b9826\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042144    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-data\") pod \"22b06a90-c616-4913-9a07-52eec95b9826\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042162    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-credential-socket\") pod \"7d541627-125c-4a48-b599-129c78b75c94\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042207    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-credential-socket\") pod \"22b06a90-c616-4913-9a07-52eec95b9826\" (UID: \"22b06a90-c616-4913-9a07-52eec95b9826\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042234    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-kube-api-access-ngkcd\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-kube-api-access-ngkcd\") pod \"7d541627-125c-4a48-b599-129c78b75c94\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042269    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-istio-token\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-istio-token\") pod \"7d541627-125c-4a48-b599-129c78b75c94\" (UID: \"7d541627-125c-4a48-b599-129c78b75c94\") "
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042707    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-envoy" pod "7d541627-125c-4a48-b599-129c78b75c94" (UID: "7d541627-125c-4a48-b599-129c78b75c94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042884    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-envoy" pod "22b06a90-c616-4913-9a07-52eec95b9826" (UID: "22b06a90-c616-4913-9a07-52eec95b9826"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042936    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042955    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042969    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417ed549-3aaa-47e6-b135-fa4317813918-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042983    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042995    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.043009    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.043085    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.043103    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdsx8\" (UniqueName: \"kubernetes.io/projected/417ed549-3aaa-47e6-b135-fa4317813918-kube-api-access-gdsx8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.043116    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.043128    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417ed549-3aaa-47e6-b135-fa4317813918-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.043140    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b06a90-c616-4913-9a07-52eec95b9826-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.042714    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d541627-125c-4a48-b599-129c78b75c94-istiod-ca-cert" pod "7d541627-125c-4a48-b599-129c78b75c94" (UID: "7d541627-125c-4a48-b599-129c78b75c94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.043165    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-data" pod "22b06a90-c616-4913-9a07-52eec95b9826" (UID: "22b06a90-c616-4913-9a07-52eec95b9826"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.044202 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.043973    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-credential-socket" pod "22b06a90-c616-4913-9a07-52eec95b9826" (UID: "22b06a90-c616-4913-9a07-52eec95b9826"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.045956 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.044948    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-socket" pod "22b06a90-c616-4913-9a07-52eec95b9826" (UID: "22b06a90-c616-4913-9a07-52eec95b9826"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.045956 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.045727    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22b06a90-c616-4913-9a07-52eec95b9826-istio-podinfo" pod "22b06a90-c616-4913-9a07-52eec95b9826" (UID: "22b06a90-c616-4913-9a07-52eec95b9826"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:55.046029 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.045997    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-istio-token" pod "22b06a90-c616-4913-9a07-52eec95b9826" (UID: "22b06a90-c616-4913-9a07-52eec95b9826"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:55.046387 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.046285    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-kube-api-access-d9zld" pod "22b06a90-c616-4913-9a07-52eec95b9826" (UID: "22b06a90-c616-4913-9a07-52eec95b9826"). InnerVolumeSpecName "kube-api-access-d9zld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:55.046387 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.046287    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-data" pod "7d541627-125c-4a48-b599-129c78b75c94" (UID: "7d541627-125c-4a48-b599-129c78b75c94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.046387 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.046350    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-credential-socket" pod "7d541627-125c-4a48-b599-129c78b75c94" (UID: "7d541627-125c-4a48-b599-129c78b75c94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:55.047803 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.047779    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-istio-token" pod "7d541627-125c-4a48-b599-129c78b75c94" (UID: "7d541627-125c-4a48-b599-129c78b75c94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:55.048158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.048045    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d541627-125c-4a48-b599-129c78b75c94-istio-podinfo" pod "7d541627-125c-4a48-b599-129c78b75c94" (UID: "7d541627-125c-4a48-b599-129c78b75c94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:55.048158 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.048082    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-kube-api-access-ngkcd" pod "7d541627-125c-4a48-b599-129c78b75c94" (UID: "7d541627-125c-4a48-b599-129c78b75c94"). InnerVolumeSpecName "kube-api-access-ngkcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:55.145252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145145    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145185    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b06a90-c616-4913-9a07-52eec95b9826-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145200    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145214    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d541627-125c-4a48-b599-129c78b75c94-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145229    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d541627-125c-4a48-b599-129c78b75c94-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145243    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9zld\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-kube-api-access-d9zld\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145260    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b06a90-c616-4913-9a07-52eec95b9826-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145274    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145288    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d541627-125c-4a48-b599-129c78b75c94-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145302    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145316    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngkcd\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-kube-api-access-ngkcd\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145329    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d541627-125c-4a48-b599-129c78b75c94-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.145705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.145343    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b06a90-c616-4913-9a07-52eec95b9826-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:55.504176 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.504136    5725 generic.go:356] "Generic (PLEG): container finished" podID="417ed549-3aaa-47e6-b135-fa4317813918" containerID="85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b" exitCode=0
Nov 01 12:50:55.504623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.504189    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"
Nov 01 12:50:55.504623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.504219    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" event={"ID":"417ed549-3aaa-47e6-b135-fa4317813918","Type":"ContainerDied","Data":"85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b"}
Nov 01 12:50:55.504623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.504275    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk" event={"ID":"417ed549-3aaa-47e6-b135-fa4317813918","Type":"ContainerDied","Data":"9ecd4702c40619dffa0aec3ba8d6d1c503300e25438c5f08cb97445d2e47c3d1"}
Nov 01 12:50:55.504623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.504290    5725 scope.go:122] "RemoveContainer" containerID="85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b"
Nov 01 12:50:55.505851 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.505827    5725 generic.go:356] "Generic (PLEG): container finished" podID="22b06a90-c616-4913-9a07-52eec95b9826" containerID="3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39" exitCode=0
Nov 01 12:50:55.505975 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.505880    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"
Nov 01 12:50:55.505975 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.505888    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" event={"ID":"22b06a90-c616-4913-9a07-52eec95b9826","Type":"ContainerDied","Data":"3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39"}
Nov 01 12:50:55.505975 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.505935    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk" event={"ID":"22b06a90-c616-4913-9a07-52eec95b9826","Type":"ContainerDied","Data":"012f64f62af08006ccfed7b7724666a6e5541c0d0975c714751d71d3d4326dd5"}
Nov 01 12:50:55.507614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.507591    5725 generic.go:356] "Generic (PLEG): container finished" podID="7d541627-125c-4a48-b599-129c78b75c94" containerID="1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56" exitCode=0
Nov 01 12:50:55.507723 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.507631    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"
Nov 01 12:50:55.507723 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.507673    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" event={"ID":"7d541627-125c-4a48-b599-129c78b75c94","Type":"ContainerDied","Data":"1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56"}
Nov 01 12:50:55.507723 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.507719    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp" event={"ID":"7d541627-125c-4a48-b599-129c78b75c94","Type":"ContainerDied","Data":"e4fc4629f2c76595fb88d64dbdbcaf8cf5e557ba665fbea4822f6d68f241045b"}
Nov 01 12:50:55.518456 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.518437    5725 scope.go:122] "RemoveContainer" containerID="85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b"
Nov 01 12:50:55.518799 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:55.518772    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b\": container with ID starting with 85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b not found: ID does not exist" containerID="85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b"
Nov 01 12:50:55.518903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.518806    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b"} err="failed to get container status \"85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b\": rpc error: code = NotFound desc = could not find container \"85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b\": container with ID starting with 85ec06645ac4c7081242fda3aac27803b3b9c00b1139f5fc3f519c7ed0251c2b not found: ID does not exist"
Nov 01 12:50:55.518903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.518824    5725 scope.go:122] "RemoveContainer" containerID="3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39"
Nov 01 12:50:55.529882 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.529859    5725 scope.go:122] "RemoveContainer" containerID="3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39"
Nov 01 12:50:55.530222 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:55.530201    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39\": container with ID starting with 3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39 not found: ID does not exist" containerID="3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39"
Nov 01 12:50:55.530325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.530233    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39"} err="failed to get container status \"3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39\": rpc error: code = NotFound desc = could not find container \"3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39\": container with ID starting with 3f51c2317cf22833ea6561503055a2f8b775d5db821119dfbfd4c62150a5db39 not found: ID does not exist"
Nov 01 12:50:55.530325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.530257    5725 scope.go:122] "RemoveContainer" containerID="1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56"
Nov 01 12:50:55.548626 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.547691    5725 scope.go:122] "RemoveContainer" containerID="1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56"
Nov 01 12:50:55.549261 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:50:55.549230    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56\": container with ID starting with 1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56 not found: ID does not exist" containerID="1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56"
Nov 01 12:50:55.549400 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.549270    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56"} err="failed to get container status \"1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56\": rpc error: code = NotFound desc = could not find container \"1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56\": container with ID starting with 1d2fd0152eefb6a182f62a22c85d5d100ade569ec8e98da33c607746dbbb7b56 not found: ID does not exist"
Nov 01 12:50:55.660965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.660908    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"]
Nov 01 12:50:55.708200 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.708166    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-qncpp"]
Nov 01 12:50:55.766488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.766379    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"]
Nov 01 12:50:55.770381 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.770355    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-zspwk"]
Nov 01 12:50:55.872875 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.872820    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"]
Nov 01 12:50:55.876724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:55.876695    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wfkrk"]
Nov 01 12:50:56.617467 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:56.617432    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b06a90-c616-4913-9a07-52eec95b9826" path="/var/lib/kubelet/pods/22b06a90-c616-4913-9a07-52eec95b9826/volumes"
Nov 01 12:50:56.617969 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:56.617953    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417ed549-3aaa-47e6-b135-fa4317813918" path="/var/lib/kubelet/pods/417ed549-3aaa-47e6-b135-fa4317813918/volumes"
Nov 01 12:50:56.618412 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:50:56.618399    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d541627-125c-4a48-b599-129c78b75c94" path="/var/lib/kubelet/pods/7d541627-125c-4a48-b599-129c78b75c94/volumes"
Nov 01 12:51:01.030007 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:51:01.029971    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:51:04.343494 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:04.343454    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"]
Nov 01 12:51:04.343929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:04.343684    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" containerID="cri-o://fd5d586fc512abb6cfbdad01ee753795787abac1c316b059fb3bbfcaaccec75c" gracePeriod=30
Nov 01 12:51:04.390895 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:04.390855    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"]
Nov 01 12:51:04.391191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:04.391161    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy" containerID="cri-o://4c3d067e319c03646741e69ed093009bb2f0e32da3cd8890d5a63f22a001cdb7" gracePeriod=30
Nov 01 12:51:04.397645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:04.397603    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"]
Nov 01 12:51:04.397916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:04.397864    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy" containerID="cri-o://bf64ed1286f90583b4af2f6fdbd9c9b3b88a1f9fa32a83e593ff8c4451b2edb8" gracePeriod=30
Nov 01 12:51:07.545790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:07.545748    5725 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:51:07.546277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:07.545810    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:51:08.586545 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.586512    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"]
Nov 01 12:51:08.587203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587155    5725 cpu_manager.go:414] "containerMap: removing container" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy"
Nov 01 12:51:08.587203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587170    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy"
Nov 01 12:51:08.587203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587195    5725 cpu_manager.go:414] "containerMap: removing container" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy"
Nov 01 12:51:08.587203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587201    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy"
Nov 01 12:51:08.587407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587216    5725 cpu_manager.go:414] "containerMap: removing container" podUID="549b3ed6-875e-4d00-b16a-c5a4feb2736d" containerName="revoke-keys"
Nov 01 12:51:08.587407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587222    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549b3ed6-875e-4d00-b16a-c5a4feb2736d" containerName="revoke-keys"
Nov 01 12:51:08.587407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587244    5725 cpu_manager.go:414] "containerMap: removing container" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy"
Nov 01 12:51:08.587407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587249    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy"
Nov 01 12:51:08.587407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587338    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="417ed549-3aaa-47e6-b135-fa4317813918" containerName="istio-proxy"
Nov 01 12:51:08.587407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587348    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d541627-125c-4a48-b599-129c78b75c94" containerName="istio-proxy"
Nov 01 12:51:08.587407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587357    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b06a90-c616-4913-9a07-52eec95b9826" containerName="istio-proxy"
Nov 01 12:51:08.587407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.587365    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="549b3ed6-875e-4d00-b16a-c5a4feb2736d" containerName="revoke-keys"
Nov 01 12:51:08.591162 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.591143    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.661546 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.661503    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.662151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.661838    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193c5d61-aea3-44c8-9d2b-0236899753b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.662151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.661953    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.662151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.662035    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwzc\" (UniqueName: \"kubernetes.io/projected/193c5d61-aea3-44c8-9d2b-0236899753b9-kube-api-access-6nwzc\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.688119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.688087    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-0cd09f-serving-cert\""
Nov 01 12:51:08.688282 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.688256    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-758k9\""
Nov 01 12:51:08.688338 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.688295    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 12:51:08.692437 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.692415    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"]
Nov 01 12:51:08.763599 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.763561    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.763878 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.763857    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwzc\" (UniqueName: \"kubernetes.io/projected/193c5d61-aea3-44c8-9d2b-0236899753b9-kube-api-access-6nwzc\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.764103 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:51:08.764071    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-serving-cert: secret "maas-api-e2e-shared-b-w1-0cd09f-serving-cert" not found
Nov 01 12:51:08.764185 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:51:08.764160    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls podName:193c5d61-aea3-44c8-9d2b-0236899753b9 nodeName:}" failed. No retries permitted until 2026-11-01 12:51:09.264137071 +0000 UTC m=+1710.771328289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls") pod "maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp" (UID: "193c5d61-aea3-44c8-9d2b-0236899753b9") : secret "maas-api-e2e-shared-b-w1-0cd09f-serving-cert" not found
Nov 01 12:51:08.764389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.764073    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.764654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.764632    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193c5d61-aea3-44c8-9d2b-0236899753b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.765310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.765289    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193c5d61-aea3-44c8-9d2b-0236899753b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.767261 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.767240    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:08.819384 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:08.819356    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwzc\" (UniqueName: \"kubernetes.io/projected/193c5d61-aea3-44c8-9d2b-0236899753b9-kube-api-access-6nwzc\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:09.274503 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.274448    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:09.276907 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.276877    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:09.505392 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.505324    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:09.642103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.641104    5725 generic.go:356] "Generic (PLEG): container finished" podID="285f8581-c11d-4354-b525-297dc8159f8a" containerID="fd5d586fc512abb6cfbdad01ee753795787abac1c316b059fb3bbfcaaccec75c" exitCode=0
Nov 01 12:51:09.642103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.641242    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" event={"ID":"285f8581-c11d-4354-b525-297dc8159f8a","Type":"ContainerDied","Data":"fd5d586fc512abb6cfbdad01ee753795787abac1c316b059fb3bbfcaaccec75c"}
Nov 01 12:51:09.643920 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.643891    5725 generic.go:356] "Generic (PLEG): container finished" podID="da118069-a12c-4401-b3ce-279be31f094c" containerID="bf64ed1286f90583b4af2f6fdbd9c9b3b88a1f9fa32a83e593ff8c4451b2edb8" exitCode=0
Nov 01 12:51:09.644064 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.643963    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" event={"ID":"da118069-a12c-4401-b3ce-279be31f094c","Type":"ContainerDied","Data":"bf64ed1286f90583b4af2f6fdbd9c9b3b88a1f9fa32a83e593ff8c4451b2edb8"}
Nov 01 12:51:09.646928 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.646900    5725 generic.go:356] "Generic (PLEG): container finished" podID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerID="4c3d067e319c03646741e69ed093009bb2f0e32da3cd8890d5a63f22a001cdb7" exitCode=0
Nov 01 12:51:09.647031 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.646982    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" event={"ID":"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b","Type":"ContainerDied","Data":"4c3d067e319c03646741e69ed093009bb2f0e32da3cd8890d5a63f22a001cdb7"}
Nov 01 12:51:09.691543 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.691517    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:51:09.785222 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.785134    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-credential-socket\") pod \"285f8581-c11d-4354-b525-297dc8159f8a\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") "
Nov 01 12:51:09.785393 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.785293    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-data\") pod \"285f8581-c11d-4354-b525-297dc8159f8a\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") "
Nov 01 12:51:09.785393 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.785342    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/285f8581-c11d-4354-b525-297dc8159f8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f8581-c11d-4354-b525-297dc8159f8a-istio-podinfo\") pod \"285f8581-c11d-4354-b525-297dc8159f8a\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") "
Nov 01 12:51:09.785393 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.785383    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-envoy\") pod \"285f8581-c11d-4354-b525-297dc8159f8a\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") "
Nov 01 12:51:09.785571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.785443    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-socket\") pod \"285f8581-c11d-4354-b525-297dc8159f8a\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") "
Nov 01 12:51:09.785626 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.785597    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285f8581-c11d-4354-b525-297dc8159f8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f8581-c11d-4354-b525-297dc8159f8a-istiod-ca-cert\") pod \"285f8581-c11d-4354-b525-297dc8159f8a\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") "
Nov 01 12:51:09.785674 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.785646    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-kube-api-access-dpxq4\") pod \"285f8581-c11d-4354-b525-297dc8159f8a\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") "
Nov 01 12:51:09.785714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.785681    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-istio-token\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-istio-token\") pod \"285f8581-c11d-4354-b525-297dc8159f8a\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") "
Nov 01 12:51:09.785772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.785756    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-certs\") pod \"285f8581-c11d-4354-b525-297dc8159f8a\" (UID: \"285f8581-c11d-4354-b525-297dc8159f8a\") "
Nov 01 12:51:09.788415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.788219    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-data" pod "285f8581-c11d-4354-b525-297dc8159f8a" (UID: "285f8581-c11d-4354-b525-297dc8159f8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.788630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.788591    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-socket" pod "285f8581-c11d-4354-b525-297dc8159f8a" (UID: "285f8581-c11d-4354-b525-297dc8159f8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.788785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.788760    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-credential-socket" pod "285f8581-c11d-4354-b525-297dc8159f8a" (UID: "285f8581-c11d-4354-b525-297dc8159f8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.789116 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.789086    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285f8581-c11d-4354-b525-297dc8159f8a-istiod-ca-cert" pod "285f8581-c11d-4354-b525-297dc8159f8a" (UID: "285f8581-c11d-4354-b525-297dc8159f8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:51:09.790384 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.790360    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-certs" pod "285f8581-c11d-4354-b525-297dc8159f8a" (UID: "285f8581-c11d-4354-b525-297dc8159f8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.791499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.791468    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-kube-api-access-dpxq4" pod "285f8581-c11d-4354-b525-297dc8159f8a" (UID: "285f8581-c11d-4354-b525-297dc8159f8a"). InnerVolumeSpecName "kube-api-access-dpxq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:09.810733 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.810690    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-envoy" pod "285f8581-c11d-4354-b525-297dc8159f8a" (UID: "285f8581-c11d-4354-b525-297dc8159f8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.812751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.812719    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-istio-token" pod "285f8581-c11d-4354-b525-297dc8159f8a" (UID: "285f8581-c11d-4354-b525-297dc8159f8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:09.814898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.814863    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/285f8581-c11d-4354-b525-297dc8159f8a-istio-podinfo" pod "285f8581-c11d-4354-b525-297dc8159f8a" (UID: "285f8581-c11d-4354-b525-297dc8159f8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:51:09.869346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.869318    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:51:09.871326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.871308    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:51:09.887406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.887370    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:09.887406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.887400    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:09.887579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.887417    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:09.887579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.887426    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:09.887579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.887436    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285f8581-c11d-4354-b525-297dc8159f8a-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:09.887579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.887445    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:09.887579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.887453    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285f8581-c11d-4354-b525-297dc8159f8a-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:09.887579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.887461    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285f8581-c11d-4354-b525-297dc8159f8a-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:09.887579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.887471    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpxq4\" (UniqueName: \"kubernetes.io/projected/285f8581-c11d-4354-b525-297dc8159f8a-kube-api-access-dpxq4\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:09.946497 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.946418    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"]
Nov 01 12:51:09.953956 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:51:09.953896    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193c5d61_aea3_44c8_9d2b_0236899753b9.slice/crio-108fe807de917f2db7618b853037517250653f40a12a9a9dee05329b561a772f WatchSource:0}: Error finding container 108fe807de917f2db7618b853037517250653f40a12a9a9dee05329b561a772f: Status 404 returned error can't find the container with id 108fe807de917f2db7618b853037517250653f40a12a9a9dee05329b561a772f
Nov 01 12:51:09.954911 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.954813    5725 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:51:09.988103 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988063    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-podinfo\") pod \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988121    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da118069-a12c-4401-b3ce-279be31f094c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da118069-a12c-4401-b3ce-279be31f094c-istiod-ca-cert\") pod \"da118069-a12c-4401-b3ce-279be31f094c\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988162    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-data\") pod \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988238    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-istio-token\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-istio-token\") pod \"da118069-a12c-4401-b3ce-279be31f094c\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988306    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-socket\") pod \"da118069-a12c-4401-b3ce-279be31f094c\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988346    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-kube-api-access-xvxxh\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-kube-api-access-xvxxh\") pod \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988382    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istiod-ca-cert\") pod \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988410    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-token\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-token\") pod \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988469    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-socket\") pod \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988493    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-data\") pod \"da118069-a12c-4401-b3ce-279be31f094c\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988531    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-certs\") pod \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988579    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-kube-api-access-kkcvl\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-kube-api-access-kkcvl\") pod \"da118069-a12c-4401-b3ce-279be31f094c\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988665    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-certs\") pod \"da118069-a12c-4401-b3ce-279be31f094c\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988694    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-envoy\") pod \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988732    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-credential-socket\") pod \"da118069-a12c-4401-b3ce-279be31f094c\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988774    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-envoy\") pod \"da118069-a12c-4401-b3ce-279be31f094c\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988842    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da118069-a12c-4401-b3ce-279be31f094c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da118069-a12c-4401-b3ce-279be31f094c-istio-podinfo\") pod \"da118069-a12c-4401-b3ce-279be31f094c\" (UID: \"da118069-a12c-4401-b3ce-279be31f094c\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.988868    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-credential-socket\") pod \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\" (UID: \"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b\") "
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.989929    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-credential-socket" pod "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" (UID: "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.989941    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-socket" pod "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" (UID: "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.990163    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-data" pod "da118069-a12c-4401-b3ce-279be31f094c" (UID: "da118069-a12c-4401-b3ce-279be31f094c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.990375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.990362    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-certs" pod "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" (UID: "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.991566 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.990394    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da118069-a12c-4401-b3ce-279be31f094c-istiod-ca-cert" pod "da118069-a12c-4401-b3ce-279be31f094c" (UID: "da118069-a12c-4401-b3ce-279be31f094c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:51:09.991566 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.990992    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-data" pod "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" (UID: "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.991566 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.991036    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-podinfo" pod "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" (UID: "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:51:09.991566 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.991321    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-credential-socket" pod "da118069-a12c-4401-b3ce-279be31f094c" (UID: "da118069-a12c-4401-b3ce-279be31f094c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.991803 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.991591    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-certs" pod "da118069-a12c-4401-b3ce-279be31f094c" (UID: "da118069-a12c-4401-b3ce-279be31f094c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.993151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.993122    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-kube-api-access-kkcvl" pod "da118069-a12c-4401-b3ce-279be31f094c" (UID: "da118069-a12c-4401-b3ce-279be31f094c"). InnerVolumeSpecName "kube-api-access-kkcvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:09.993472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.993446    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-istio-token" pod "da118069-a12c-4401-b3ce-279be31f094c" (UID: "da118069-a12c-4401-b3ce-279be31f094c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:09.993472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.993461    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istiod-ca-cert" pod "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" (UID: "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:51:09.993757 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.993732    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-socket" pod "da118069-a12c-4401-b3ce-279be31f094c" (UID: "da118069-a12c-4401-b3ce-279be31f094c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.994450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.994411    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-envoy" pod "da118069-a12c-4401-b3ce-279be31f094c" (UID: "da118069-a12c-4401-b3ce-279be31f094c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.994712 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.994690    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-envoy" pod "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" (UID: "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:09.995615 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.995593    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-kube-api-access-xvxxh" pod "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" (UID: "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b"). InnerVolumeSpecName "kube-api-access-xvxxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:09.995721 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.995646    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-token" pod "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" (UID: "b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:09.996250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:09.996225    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da118069-a12c-4401-b3ce-279be31f094c-istio-podinfo" pod "da118069-a12c-4401-b3ce-279be31f094c" (UID: "da118069-a12c-4401-b3ce-279be31f094c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:51:10.090833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090752    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.090833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090793    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da118069-a12c-4401-b3ce-279be31f094c-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.090833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090803    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.090833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090811    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.090833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090822    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.090833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090830    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvxxh\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-kube-api-access-xvxxh\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090840    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090848    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090856    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090864    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090873    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090881    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkcvl\" (UniqueName: \"kubernetes.io/projected/da118069-a12c-4401-b3ce-279be31f094c-kube-api-access-kkcvl\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090889    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090897    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090905    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090913    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da118069-a12c-4401-b3ce-279be31f094c-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090920    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da118069-a12c-4401-b3ce-279be31f094c-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.091167 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.090928    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.655992 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.655965    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"
Nov 01 12:51:10.656401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.655962    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb" event={"ID":"285f8581-c11d-4354-b525-297dc8159f8a","Type":"ContainerDied","Data":"0aa17070bbebde2a8cbf5490e0dabfaf0f0ebe07083a4db4bd18d2295e9542c6"}
Nov 01 12:51:10.656401 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.656116    5725 scope.go:122] "RemoveContainer" containerID="fd5d586fc512abb6cfbdad01ee753795787abac1c316b059fb3bbfcaaccec75c"
Nov 01 12:51:10.657718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.657596    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"
Nov 01 12:51:10.657718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.657598    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww" event={"ID":"da118069-a12c-4401-b3ce-279be31f094c","Type":"ContainerDied","Data":"96f76a5dc4909801d59a1528780f7444abaf7a9620ca8820d353eddae513e87d"}
Nov 01 12:51:10.659347 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.659322    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp" event={"ID":"193c5d61-aea3-44c8-9d2b-0236899753b9","Type":"ContainerStarted","Data":"108fe807de917f2db7618b853037517250653f40a12a9a9dee05329b561a772f"}
Nov 01 12:51:10.660906 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.660888    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"
Nov 01 12:51:10.660992 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.660917    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc" event={"ID":"b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b","Type":"ContainerDied","Data":"9b5384c39c39804f86790debd958a86e002279d7b6c93f3a603e478950d33fb6"}
Nov 01 12:51:10.668164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.668147    5725 scope.go:122] "RemoveContainer" containerID="bf64ed1286f90583b4af2f6fdbd9c9b3b88a1f9fa32a83e593ff8c4451b2edb8"
Nov 01 12:51:10.678043 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.678023    5725 scope.go:122] "RemoveContainer" containerID="4c3d067e319c03646741e69ed093009bb2f0e32da3cd8890d5a63f22a001cdb7"
Nov 01 12:51:10.814397 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.814363    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"]
Nov 01 12:51:10.860529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.860487    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-hpxjc"]
Nov 01 12:51:10.941283 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.934458    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"]
Nov 01 12:51:10.953072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:10.941882    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-tlxgb"]
Nov 01 12:51:11.038482 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:11.037393    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"]
Nov 01 12:51:11.047120 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:11.047085    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-xjnww"]
Nov 01 12:51:12.619570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:12.619492    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285f8581-c11d-4354-b525-297dc8159f8a" path="/var/lib/kubelet/pods/285f8581-c11d-4354-b525-297dc8159f8a/volumes"
Nov 01 12:51:12.620061 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:12.620035    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" path="/var/lib/kubelet/pods/b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b/volumes"
Nov 01 12:51:12.620518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:12.620504    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da118069-a12c-4401-b3ce-279be31f094c" path="/var/lib/kubelet/pods/da118069-a12c-4401-b3ce-279be31f094c/volumes"
Nov 01 12:51:12.685804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:12.685765    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp" event={"ID":"193c5d61-aea3-44c8-9d2b-0236899753b9","Type":"ContainerStarted","Data":"8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c"}
Nov 01 12:51:12.685981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:12.685834    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:12.794690 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:12.794632    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp" podStartSLOduration=2.521183378 podStartE2EDuration="4.794612592s" totalImagesPullingTime="2.273429214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:51:12.793836785 +0000 UTC m=+1714.301028024" watchObservedRunningTime="2026-11-01 12:51:12.794612592 +0000 UTC m=+1714.301803809"
Nov 01 12:51:18.700039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:18.700010    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:51:31.041377 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:51:31.041334    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:51:34.619115 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:34.619075    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"]
Nov 01 12:51:34.619655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:34.619288    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy" containerID="cri-o://027a9e92548b3613afc1a8138240b3173bd4c11dd0bfc9815a853a0ea17a4116" gracePeriod=30
Nov 01 12:51:34.660325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:34.660285    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"]
Nov 01 12:51:34.660795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:34.660742    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy" containerID="cri-o://cf829ff5ba9ba77dc13e64f54e3b4528f403d78c4711dbc999e51dc13a2b9e29" gracePeriod=30
Nov 01 12:51:34.668603 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:34.668576    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"]
Nov 01 12:51:34.669007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:34.668963    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy" containerID="cri-o://93b56ee831d68a33c28d36aa93460143088e1e88a3a2a879ed5924de9a4c49e4" gracePeriod=30
Nov 01 12:51:35.946797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:35.946758    5725 generic.go:356] "Generic (PLEG): container finished" podID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerID="c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2" exitCode=0
Nov 01 12:51:35.947497 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:35.946803    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" event={"ID":"c0b7d530-db61-4ec9-9647-deb92b22a82f","Type":"ContainerDied","Data":"c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2"}
Nov 01 12:51:35.948203 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:35.948178    5725 scope.go:122] "RemoveContainer" containerID="c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2"
Nov 01 12:51:35.950373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:35.950332    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:51:35.950461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:35.950426    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:51:35.950503 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:35.950478    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:51:36.787621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:36.787581    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:51:36.972073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:36.971554    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" event={"ID":"c0b7d530-db61-4ec9-9647-deb92b22a82f","Type":"ContainerStarted","Data":"c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8"}
Nov 01 12:51:37.948074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:37.946756    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:51:37.948074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:37.946924    5725 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 12:51:37.948074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:37.946968    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 12:51:38.291188 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:38.291145    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:51:38.317209 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:38.317160    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:51:38.456075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:38.446692    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:51:38.464511 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:38.464471    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:51:38.732136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:38.731740    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"]
Nov 01 12:51:38.732136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:38.731798    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ffb38d-nc5br"]
Nov 01 12:51:38.971074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:38.947156    5725 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 12:51:38.971074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:38.947227    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 12:51:39.535074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.534298    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"]
Nov 01 12:51:39.535698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.535660    5725 cpu_manager.go:414] "containerMap: removing container" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy"
Nov 01 12:51:39.535896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.535881    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy"
Nov 01 12:51:39.536027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.536016    5725 cpu_manager.go:414] "containerMap: removing container" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy"
Nov 01 12:51:39.536145 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.536131    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy"
Nov 01 12:51:39.536271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.536260    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy"
Nov 01 12:51:39.536375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.536363    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy"
Nov 01 12:51:39.536734 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.536719    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="285f8581-c11d-4354-b525-297dc8159f8a" containerName="istio-proxy"
Nov 01 12:51:39.536870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.536856    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="da118069-a12c-4401-b3ce-279be31f094c" containerName="istio-proxy"
Nov 01 12:51:39.536974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.536963    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4447d89-8ce2-4fc8-88e5-8abad0d8bd4b" containerName="istio-proxy"
Nov 01 12:51:39.540623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.540601    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.592259 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.592163    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"]
Nov 01 12:51:39.597375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.596641    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-envoy\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.597375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.596739    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-credential-socket\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.597375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.596851    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-certs\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.597375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.596979    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-socket\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.597375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.597037    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-podinfo\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.597375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.597084    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istiod-ca-cert\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.597375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.597118    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kltd8\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-kube-api-access-kltd8\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.597375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.597156    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-data\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.597375 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.597180    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-token\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.630555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.630518    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-6124f9-openshift-default-dockercfg-pbwhl\""
Nov 01 12:51:39.649365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.645283    5725 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:51:39.649365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.645365    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:51:39.699126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.698388    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-envoy\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.699126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.698493    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-credential-socket\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.699126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.698617    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-certs\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.699126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.698752    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-socket\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.699126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.698816    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-podinfo\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.699126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.698849    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istiod-ca-cert\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.699126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.698883    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kltd8\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-kube-api-access-kltd8\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.699126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.698919    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-data\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.699126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.698944    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-token\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.705065 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.703242    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istiod-ca-cert\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.705065 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.703486    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-data\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.705065 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.703700    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-certs\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.705065 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.703912    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-credential-socket\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.705065 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.704175    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-socket\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.892823 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.855066    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-podinfo\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.915447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.914827    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-envoy\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.916643 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.916611    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kltd8\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-kube-api-access-kltd8\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:39.919442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:39.919386    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-token\") pod \"e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") " pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:40.041719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.033437    5725 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 12:51:40.041719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.033501    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 12:51:40.183340 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.179290    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"]
Nov 01 12:51:40.183340 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.180299    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:40.189172 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.183540    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58e0934-lwzsc"]
Nov 01 12:51:40.219527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.219490    5725 scope.go:122] "RemoveContainer" containerID="66917f7e8cfe097221d803fe9f234579de0f07037f524de23f7590cb7359fac8"
Nov 01 12:51:40.226715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.226683    5725 generic.go:356] "Generic (PLEG): container finished" podID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerID="027a9e92548b3613afc1a8138240b3173bd4c11dd0bfc9815a853a0ea17a4116" exitCode=0
Nov 01 12:51:40.226927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.226907    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" event={"ID":"30185b0c-125b-45f0-b1fa-14af84e9a913","Type":"ContainerDied","Data":"027a9e92548b3613afc1a8138240b3173bd4c11dd0bfc9815a853a0ea17a4116"}
Nov 01 12:51:40.232359 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.231675    5725 generic.go:356] "Generic (PLEG): container finished" podID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerID="cf829ff5ba9ba77dc13e64f54e3b4528f403d78c4711dbc999e51dc13a2b9e29" exitCode=0
Nov 01 12:51:40.232359 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.231789    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" event={"ID":"8f43f391-f98e-45ff-8fdc-ea887e3b5115","Type":"ContainerDied","Data":"cf829ff5ba9ba77dc13e64f54e3b4528f403d78c4711dbc999e51dc13a2b9e29"}
Nov 01 12:51:40.237002 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.236603    5725 generic.go:356] "Generic (PLEG): container finished" podID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerID="93b56ee831d68a33c28d36aa93460143088e1e88a3a2a879ed5924de9a4c49e4" exitCode=0
Nov 01 12:51:40.237002 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.236752    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" event={"ID":"6d18b955-e42c-4c05-92ba-f1ab13f98811","Type":"ContainerDied","Data":"93b56ee831d68a33c28d36aa93460143088e1e88a3a2a879ed5924de9a4c49e4"}
Nov 01 12:51:40.302654 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.302624    5725 scope.go:122] "RemoveContainer" containerID="4d499e3d8eaa4d814098b6f53e45b8ff8be4f6e5796b57db5ebfc0958380d445"
Nov 01 12:51:40.320287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.320260    5725 scope.go:122] "RemoveContainer" containerID="70a0f3a15cd74fd6a69856feb3ddf693953b26f8ee136e64f65c30c33966e8b3"
Nov 01 12:51:40.339226 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.339067    5725 scope.go:122] "RemoveContainer" containerID="42078165e4726257dd150f0983dcd38dacb7f5698a78df3b05e83f8269688e75"
Nov 01 12:51:40.411944 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.409944    5725 scope.go:122] "RemoveContainer" containerID="52ffa171bedc02bc30ddfeff4fc622d48740b01616336501bf4a2a11df3b90fb"
Nov 01 12:51:40.429133 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.428739    5725 scope.go:122] "RemoveContainer" containerID="d344b3418ebd8bb74f352488f04391ee12a579cf386d0e9515308f7b901fce2a"
Nov 01 12:51:40.956149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.955209    5725 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 12:51:40.956149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.955265    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 12:51:40.996118 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.992223    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b709ad8-e158-43ff-a23e-189629834ef2" path="/var/lib/kubelet/pods/0b709ad8-e158-43ff-a23e-189629834ef2/volumes"
Nov 01 12:51:40.996118 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:40.992872    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549b3ed6-875e-4d00-b16a-c5a4feb2736d" path="/var/lib/kubelet/pods/549b3ed6-875e-4d00-b16a-c5a4feb2736d/volumes"
Nov 01 12:51:41.063318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.063280    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:51:41.090547 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.090507    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:51:41.106706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.106675    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:51:41.193315 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193281    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-envoy\") pod \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") "
Nov 01 12:51:41.193461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193348    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-certs\") pod \"30185b0c-125b-45f0-b1fa-14af84e9a913\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") "
Nov 01 12:51:41.193461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193399    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-certs\") pod \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") "
Nov 01 12:51:41.193461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193432    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-data\") pod \"30185b0c-125b-45f0-b1fa-14af84e9a913\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") "
Nov 01 12:51:41.193636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193464    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istiod-ca-cert\") pod \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") "
Nov 01 12:51:41.193636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193514    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d18b955-e42c-4c05-92ba-f1ab13f98811-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d18b955-e42c-4c05-92ba-f1ab13f98811-istiod-ca-cert\") pod \"6d18b955-e42c-4c05-92ba-f1ab13f98811\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") "
Nov 01 12:51:41.193636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193552    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-podinfo\") pod \"30185b0c-125b-45f0-b1fa-14af84e9a913\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") "
Nov 01 12:51:41.193636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193580    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-data\") pod \"6d18b955-e42c-4c05-92ba-f1ab13f98811\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") "
Nov 01 12:51:41.193636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193617    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-certs\") pod \"6d18b955-e42c-4c05-92ba-f1ab13f98811\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") "
Nov 01 12:51:41.193924 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193666    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-envoy\") pod \"6d18b955-e42c-4c05-92ba-f1ab13f98811\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") "
Nov 01 12:51:41.193924 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193690    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-credential-socket\") pod \"6d18b955-e42c-4c05-92ba-f1ab13f98811\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") "
Nov 01 12:51:41.193924 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193722    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-certs" pod "30185b0c-125b-45f0-b1fa-14af84e9a913" (UID: "30185b0c-125b-45f0-b1fa-14af84e9a913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.193924 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193906    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istiod-ca-cert" pod "8f43f391-f98e-45ff-8fdc-ea887e3b5115" (UID: "8f43f391-f98e-45ff-8fdc-ea887e3b5115"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:51:41.194209 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194128    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-certs" pod "8f43f391-f98e-45ff-8fdc-ea887e3b5115" (UID: "8f43f391-f98e-45ff-8fdc-ea887e3b5115"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.194472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194318    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-data" pod "30185b0c-125b-45f0-b1fa-14af84e9a913" (UID: "30185b0c-125b-45f0-b1fa-14af84e9a913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.194472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.193731    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-kube-api-access-kgx57\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-kube-api-access-kgx57\") pod \"30185b0c-125b-45f0-b1fa-14af84e9a913\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") "
Nov 01 12:51:41.194472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194415    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-data\") pod \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") "
Nov 01 12:51:41.194678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194535    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-token\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-token\") pod \"30185b0c-125b-45f0-b1fa-14af84e9a913\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") "
Nov 01 12:51:41.194678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194572    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-socket\") pod \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") "
Nov 01 12:51:41.194678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194616    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-token\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-token\") pod \"6d18b955-e42c-4c05-92ba-f1ab13f98811\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") "
Nov 01 12:51:41.194855 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194680    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d18b955-e42c-4c05-92ba-f1ab13f98811-istiod-ca-cert" pod "6d18b955-e42c-4c05-92ba-f1ab13f98811" (UID: "6d18b955-e42c-4c05-92ba-f1ab13f98811"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:51:41.194855 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194687    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-credential-socket\") pod \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") "
Nov 01 12:51:41.194855 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194741    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-token\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-token\") pod \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") "
Nov 01 12:51:41.194855 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194788    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-credential-socket\") pod \"30185b0c-125b-45f0-b1fa-14af84e9a913\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") "
Nov 01 12:51:41.194855 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194840    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-kube-api-access-gp5k4\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-kube-api-access-gp5k4\") pod \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") "
Nov 01 12:51:41.195155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194879    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-podinfo\") pod \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\" (UID: \"8f43f391-f98e-45ff-8fdc-ea887e3b5115\") "
Nov 01 12:51:41.195155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194917    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-podinfo\") pod \"6d18b955-e42c-4c05-92ba-f1ab13f98811\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") "
Nov 01 12:51:41.195155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194950    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30185b0c-125b-45f0-b1fa-14af84e9a913-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30185b0c-125b-45f0-b1fa-14af84e9a913-istiod-ca-cert\") pod \"30185b0c-125b-45f0-b1fa-14af84e9a913\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") "
Nov 01 12:51:41.195155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.194993    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-kube-api-access-9wb47\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-kube-api-access-9wb47\") pod \"6d18b955-e42c-4c05-92ba-f1ab13f98811\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") "
Nov 01 12:51:41.195155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.195026    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-socket\") pod \"6d18b955-e42c-4c05-92ba-f1ab13f98811\" (UID: \"6d18b955-e42c-4c05-92ba-f1ab13f98811\") "
Nov 01 12:51:41.195155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.195035    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-credential-socket" pod "6d18b955-e42c-4c05-92ba-f1ab13f98811" (UID: "6d18b955-e42c-4c05-92ba-f1ab13f98811"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.195155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.195105    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-envoy\") pod \"30185b0c-125b-45f0-b1fa-14af84e9a913\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") "
Nov 01 12:51:41.195155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.195149    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-socket\") pod \"30185b0c-125b-45f0-b1fa-14af84e9a913\" (UID: \"30185b0c-125b-45f0-b1fa-14af84e9a913\") "
Nov 01 12:51:41.195915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.195612    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-credential-socket" pod "8f43f391-f98e-45ff-8fdc-ea887e3b5115" (UID: "8f43f391-f98e-45ff-8fdc-ea887e3b5115"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.195915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.195680    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-data" pod "6d18b955-e42c-4c05-92ba-f1ab13f98811" (UID: "6d18b955-e42c-4c05-92ba-f1ab13f98811"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.195915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.195884    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-socket" pod "6d18b955-e42c-4c05-92ba-f1ab13f98811" (UID: "6d18b955-e42c-4c05-92ba-f1ab13f98811"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.197660 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.195215    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-certs" pod "6d18b955-e42c-4c05-92ba-f1ab13f98811" (UID: "6d18b955-e42c-4c05-92ba-f1ab13f98811"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.198772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.198744    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-kube-api-access-kgx57" pod "30185b0c-125b-45f0-b1fa-14af84e9a913" (UID: "30185b0c-125b-45f0-b1fa-14af84e9a913"). InnerVolumeSpecName "kube-api-access-kgx57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:41.199152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199115    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-podinfo" pod "30185b0c-125b-45f0-b1fa-14af84e9a913" (UID: "30185b0c-125b-45f0-b1fa-14af84e9a913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:51:41.199892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199863    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.199892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199886    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199896    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199913    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199923    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199932    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199941    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d18b955-e42c-4c05-92ba-f1ab13f98811-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199952    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199963    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199971    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199980    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.200068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.199992    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgx57\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-kube-api-access-kgx57\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.202557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.202250    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-data" pod "8f43f391-f98e-45ff-8fdc-ea887e3b5115" (UID: "8f43f391-f98e-45ff-8fdc-ea887e3b5115"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.203127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.202756    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-socket" pod "8f43f391-f98e-45ff-8fdc-ea887e3b5115" (UID: "8f43f391-f98e-45ff-8fdc-ea887e3b5115"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.203127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.195578    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-credential-socket" pod "30185b0c-125b-45f0-b1fa-14af84e9a913" (UID: "30185b0c-125b-45f0-b1fa-14af84e9a913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.203939 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.203777    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30185b0c-125b-45f0-b1fa-14af84e9a913-istiod-ca-cert" pod "30185b0c-125b-45f0-b1fa-14af84e9a913" (UID: "30185b0c-125b-45f0-b1fa-14af84e9a913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:51:41.206307 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.205743    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-socket" pod "30185b0c-125b-45f0-b1fa-14af84e9a913" (UID: "30185b0c-125b-45f0-b1fa-14af84e9a913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.219290 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.219250    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-envoy" pod "6d18b955-e42c-4c05-92ba-f1ab13f98811" (UID: "6d18b955-e42c-4c05-92ba-f1ab13f98811"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.219481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.219452    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-envoy" pod "8f43f391-f98e-45ff-8fdc-ea887e3b5115" (UID: "8f43f391-f98e-45ff-8fdc-ea887e3b5115"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.219534 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.219259    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-envoy" pod "30185b0c-125b-45f0-b1fa-14af84e9a913" (UID: "30185b0c-125b-45f0-b1fa-14af84e9a913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:41.219724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.219694    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-podinfo" pod "6d18b955-e42c-4c05-92ba-f1ab13f98811" (UID: "6d18b955-e42c-4c05-92ba-f1ab13f98811"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:51:41.219893 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.219868    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-kube-api-access-9wb47" pod "6d18b955-e42c-4c05-92ba-f1ab13f98811" (UID: "6d18b955-e42c-4c05-92ba-f1ab13f98811"). InnerVolumeSpecName "kube-api-access-9wb47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:41.220838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.220812    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-token" pod "6d18b955-e42c-4c05-92ba-f1ab13f98811" (UID: "6d18b955-e42c-4c05-92ba-f1ab13f98811"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:41.220948 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.220871    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-token" pod "30185b0c-125b-45f0-b1fa-14af84e9a913" (UID: "30185b0c-125b-45f0-b1fa-14af84e9a913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:41.221007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.220972    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-token" pod "8f43f391-f98e-45ff-8fdc-ea887e3b5115" (UID: "8f43f391-f98e-45ff-8fdc-ea887e3b5115"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:41.222426 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.222331    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-podinfo" pod "8f43f391-f98e-45ff-8fdc-ea887e3b5115" (UID: "8f43f391-f98e-45ff-8fdc-ea887e3b5115"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:51:41.222557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.222531    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-kube-api-access-gp5k4" pod "8f43f391-f98e-45ff-8fdc-ea887e3b5115" (UID: "8f43f391-f98e-45ff-8fdc-ea887e3b5115"). InnerVolumeSpecName "kube-api-access-gp5k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:41.277184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.277141    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj" event={"ID":"8f43f391-f98e-45ff-8fdc-ea887e3b5115","Type":"ContainerDied","Data":"f07f3ba9176a0e07f2860de75e54532fc3800de73e3d52774e5d9a26bcd535e5"}
Nov 01 12:51:41.277372 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.277259    5725 scope.go:122] "RemoveContainer" containerID="cf829ff5ba9ba77dc13e64f54e3b4528f403d78c4711dbc999e51dc13a2b9e29"
Nov 01 12:51:41.277454 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.277428    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"
Nov 01 12:51:41.286508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.286477    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"
Nov 01 12:51:41.293693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.293656    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64" event={"ID":"6d18b955-e42c-4c05-92ba-f1ab13f98811","Type":"ContainerDied","Data":"94ad9d75fcab42fb056ba24965b9a91ec7b8309c74c0d76e338aa1bdf2fd0163"}
Nov 01 12:51:41.297597 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.297561    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs" event={"ID":"30185b0c-125b-45f0-b1fa-14af84e9a913","Type":"ContainerDied","Data":"665d49c98a2fd20f1a300cfd75e6817b83938d19888e8588e603ee5afcb54d2d"}
Nov 01 12:51:41.297729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.297711    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305355    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305384    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305402    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305416    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305431    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305446    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp5k4\" (UniqueName: \"kubernetes.io/projected/8f43f391-f98e-45ff-8fdc-ea887e3b5115-kube-api-access-gp5k4\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305461    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305474    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305488    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30185b0c-125b-45f0-b1fa-14af84e9a913-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305503    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wb47\" (UniqueName: \"kubernetes.io/projected/6d18b955-e42c-4c05-92ba-f1ab13f98811-kube-api-access-9wb47\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305517    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305531    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30185b0c-125b-45f0-b1fa-14af84e9a913-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305545    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305561    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d18b955-e42c-4c05-92ba-f1ab13f98811-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.306112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.305575    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f43f391-f98e-45ff-8fdc-ea887e3b5115-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:41.315356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.315331    5725 scope.go:122] "RemoveContainer" containerID="93b56ee831d68a33c28d36aa93460143088e1e88a3a2a879ed5924de9a4c49e4"
Nov 01 12:51:41.332915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.332891    5725 scope.go:122] "RemoveContainer" containerID="027a9e92548b3613afc1a8138240b3173bd4c11dd0bfc9815a853a0ea17a4116"
Nov 01 12:51:41.370260 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.370225    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"]
Nov 01 12:51:41.384105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.382793    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:51:41.384105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.382888    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:51:41.384105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.382944    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:51:41.600304 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.598616    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"]
Nov 01 12:51:41.605279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.605247    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-268qs"]
Nov 01 12:51:41.688689 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.688617    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"]
Nov 01 12:51:41.695537 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.695490    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9tcfj"]
Nov 01 12:51:41.801274 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.801245    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"]
Nov 01 12:51:41.809004 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.808964    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-9qg64"]
Nov 01 12:51:41.946074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.945510    5725 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 12:51:41.946074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:41.945571    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 12:51:42.312227 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:42.312187    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" event={"ID":"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9","Type":"ContainerStarted","Data":"e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a"}
Nov 01 12:51:42.312730 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:42.312256    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" event={"ID":"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9","Type":"ContainerStarted","Data":"d8ad0979c19fe0a0787a1bb6a29e93d4dee12cb3dc6c7eef43eaa11ee490b52f"}
Nov 01 12:51:42.431076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:42.427621    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" podStartSLOduration=3.427602031 podStartE2EDuration="3.427602031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:51:42.426189712 +0000 UTC m=+1743.933380930" watchObservedRunningTime="2026-11-01 12:51:42.427602031 +0000 UTC m=+1743.934793248"
Nov 01 12:51:42.620138 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:42.620040    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" path="/var/lib/kubelet/pods/30185b0c-125b-45f0-b1fa-14af84e9a913/volumes"
Nov 01 12:51:42.620686 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:42.620670    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" path="/var/lib/kubelet/pods/6d18b955-e42c-4c05-92ba-f1ab13f98811/volumes"
Nov 01 12:51:42.621188 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:42.621174    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" path="/var/lib/kubelet/pods/8f43f391-f98e-45ff-8fdc-ea887e3b5115/volumes"
Nov 01 12:51:42.945690 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:42.945586    5725 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 12:51:42.945877 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:42.945678    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 12:51:43.183615 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:43.183579    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:43.192217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:43.192172    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:43.321471 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:43.321439    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:43.322678 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:43.322658    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:51:43.950549 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:43.950512    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:51:43.950954 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:43.950934    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:51:43.951827 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:43.951803    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064002    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"]
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064598    5725 cpu_manager.go:414] "containerMap: removing container" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy"
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064612    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy"
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064627    5725 cpu_manager.go:414] "containerMap: removing container" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy"
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064633    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy"
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064652    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy"
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064658    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy"
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064761    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d18b955-e42c-4c05-92ba-f1ab13f98811" containerName="istio-proxy"
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064770    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="30185b0c-125b-45f0-b1fa-14af84e9a913" containerName="istio-proxy"
Nov 01 12:51:50.065072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.064779    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f43f391-f98e-45ff-8fdc-ea887e3b5115" containerName="istio-proxy"
Nov 01 12:51:50.068020 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.067992    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:51:50.118606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.118573    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"]
Nov 01 12:51:50.161072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.160985    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-6124f9\""
Nov 01 12:51:50.161239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.161129    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-6124f9-dockercfg-l972m\""
Nov 01 12:51:50.164953 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.164891    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-kube-api-access-h29rw\") pod \"payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9\" (UID: \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:51:50.165494 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.165400    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9\" (UID: \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:51:50.231285 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.231249    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"]
Nov 01 12:51:50.235371 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.235344    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:51:50.267325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.267286    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-kube-api-access-h29rw\") pod \"payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9\" (UID: \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:51:50.267606 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.267588    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9\" (UID: \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:51:50.268728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.268699    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9\" (UID: \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:51:50.292367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.292330    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"]
Nov 01 12:51:50.357134 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.348439    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-kube-api-access-h29rw\") pod \"payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9\" (UID: \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:51:50.369512 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.369481    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr\" (UID: \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:51:50.369638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.369553    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr9xr\" (UniqueName: \"kubernetes.io/projected/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-kube-api-access-tr9xr\") pod \"payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr\" (UID: \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:51:50.397121 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.394145    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:51:50.476072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.473739    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr\" (UID: \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:51:50.476072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.473834    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9xr\" (UniqueName: \"kubernetes.io/projected/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-kube-api-access-tr9xr\") pod \"payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr\" (UID: \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:51:50.476072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.475875    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr\" (UID: \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:51:50.541073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.533466    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9xr\" (UniqueName: \"kubernetes.io/projected/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-kube-api-access-tr9xr\") pod \"payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr\" (UID: \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:51:50.558778 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.557451    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:51:50.898393 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.898320    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"]
Nov 01 12:51:50.904203 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:51:50.904173    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8d1a41_d6cc_4243_ad9b_6b3c6efa2689.slice/crio-9ba0bf5b723c32625393e3a6c03a4f7e97492b5f297938cf588374c60a1b2207 WatchSource:0}: Error finding container 9ba0bf5b723c32625393e3a6c03a4f7e97492b5f297938cf588374c60a1b2207: Status 404 returned error can't find the container with id 9ba0bf5b723c32625393e3a6c03a4f7e97492b5f297938cf588374c60a1b2207
Nov 01 12:51:50.999326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:50.999293    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"]
Nov 01 12:51:51.004032 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:51:51.004002    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6685a3f8_4bf1_485f_a6bc_8e9eb0f1cd85.slice/crio-515931271e0d8e986a3c2e946fdb08455e94e19747095068ffd18d6ba8318421 WatchSource:0}: Error finding container 515931271e0d8e986a3c2e946fdb08455e94e19747095068ffd18d6ba8318421: Status 404 returned error can't find the container with id 515931271e0d8e986a3c2e946fdb08455e94e19747095068ffd18d6ba8318421
Nov 01 12:51:51.428523 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:51.428483    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr" event={"ID":"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85","Type":"ContainerStarted","Data":"cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62"}
Nov 01 12:51:51.428997 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:51.428551    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr" event={"ID":"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85","Type":"ContainerStarted","Data":"515931271e0d8e986a3c2e946fdb08455e94e19747095068ffd18d6ba8318421"}
Nov 01 12:51:51.428997 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:51.428580    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:51:51.430556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:51.430527    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9" event={"ID":"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689","Type":"ContainerStarted","Data":"68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e"}
Nov 01 12:51:51.430683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:51.430578    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:51:51.430683 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:51.430596    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9" event={"ID":"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689","Type":"ContainerStarted","Data":"9ba0bf5b723c32625393e3a6c03a4f7e97492b5f297938cf588374c60a1b2207"}
Nov 01 12:51:51.539230 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:51.539174    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr" podStartSLOduration=1.53915831 podStartE2EDuration="1.53915831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:51:51.535921752 +0000 UTC m=+1753.043112970" watchObservedRunningTime="2026-11-01 12:51:51.53915831 +0000 UTC m=+1753.046349526"
Nov 01 12:51:51.645396 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:51.645333    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9" podStartSLOduration=1.645312864 podStartE2EDuration="1.645312864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:51:51.644140517 +0000 UTC m=+1753.151331734" watchObservedRunningTime="2026-11-01 12:51:51.645312864 +0000 UTC m=+1753.152504082"
Nov 01 12:51:57.304493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.304458    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"]
Nov 01 12:51:57.307828 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.307800    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:51:57.363925 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.363893    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"]
Nov 01 12:51:57.379370 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.379336    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7dmw\" (UniqueName: \"kubernetes.io/projected/57a2e157-c261-483a-b537-48292fbb16e5-kube-api-access-k7dmw\") pod \"payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg\" (UID: \"57a2e157-c261-483a-b537-48292fbb16e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:51:57.379582 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.379550    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2e157-c261-483a-b537-48292fbb16e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg\" (UID: \"57a2e157-c261-483a-b537-48292fbb16e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:51:57.436075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.416208    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-94c13a-dockercfg-44f6r\""
Nov 01 12:51:57.436075 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.416628    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-94c13a\""
Nov 01 12:51:57.504136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.503930    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2e157-c261-483a-b537-48292fbb16e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg\" (UID: \"57a2e157-c261-483a-b537-48292fbb16e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:51:57.504347 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.504177    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7dmw\" (UniqueName: \"kubernetes.io/projected/57a2e157-c261-483a-b537-48292fbb16e5-kube-api-access-k7dmw\") pod \"payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg\" (UID: \"57a2e157-c261-483a-b537-48292fbb16e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:51:57.536073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.505550    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2e157-c261-483a-b537-48292fbb16e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg\" (UID: \"57a2e157-c261-483a-b537-48292fbb16e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:51:57.576796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.576676    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7dmw\" (UniqueName: \"kubernetes.io/projected/57a2e157-c261-483a-b537-48292fbb16e5-kube-api-access-k7dmw\") pod \"payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg\" (UID: \"57a2e157-c261-483a-b537-48292fbb16e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:51:57.597872 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.597830    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"]
Nov 01 12:51:57.605517 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.604025    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:51:57.614471 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.612986    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"]
Nov 01 12:51:57.622014 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.621984    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:51:57.711214 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.711178    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca93cb82-56a8-4a3f-806d-674a21c9d346-plugins-config-volume\") pod \"payload-processing-e2e-api-a-94c13a-649d679477-l9j2b\" (UID: \"ca93cb82-56a8-4a3f-806d-674a21c9d346\") " pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:51:57.711363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.711330    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5f9l\" (UniqueName: \"kubernetes.io/projected/ca93cb82-56a8-4a3f-806d-674a21c9d346-kube-api-access-c5f9l\") pod \"payload-processing-e2e-api-a-94c13a-649d679477-l9j2b\" (UID: \"ca93cb82-56a8-4a3f-806d-674a21c9d346\") " pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:51:57.812966 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.812933    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5f9l\" (UniqueName: \"kubernetes.io/projected/ca93cb82-56a8-4a3f-806d-674a21c9d346-kube-api-access-c5f9l\") pod \"payload-processing-e2e-api-a-94c13a-649d679477-l9j2b\" (UID: \"ca93cb82-56a8-4a3f-806d-674a21c9d346\") " pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:51:57.813197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.813180    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca93cb82-56a8-4a3f-806d-674a21c9d346-plugins-config-volume\") pod \"payload-processing-e2e-api-a-94c13a-649d679477-l9j2b\" (UID: \"ca93cb82-56a8-4a3f-806d-674a21c9d346\") " pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:51:57.813878 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.813858    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca93cb82-56a8-4a3f-806d-674a21c9d346-plugins-config-volume\") pod \"payload-processing-e2e-api-a-94c13a-649d679477-l9j2b\" (UID: \"ca93cb82-56a8-4a3f-806d-674a21c9d346\") " pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:51:57.867974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.867902    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5f9l\" (UniqueName: \"kubernetes.io/projected/ca93cb82-56a8-4a3f-806d-674a21c9d346-kube-api-access-c5f9l\") pod \"payload-processing-e2e-api-a-94c13a-649d679477-l9j2b\" (UID: \"ca93cb82-56a8-4a3f-806d-674a21c9d346\") " pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:51:57.927737 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:57.927701    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:51:58.016320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.013938    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"]
Nov 01 12:51:58.017523 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:51:58.016738    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a2e157_c261_483a_b537_48292fbb16e5.slice/crio-64de82a1c9058a4de7ddaec401e15ef29d85131b8f49d1b77adaec11966ee77e WatchSource:0}: Error finding container 64de82a1c9058a4de7ddaec401e15ef29d85131b8f49d1b77adaec11966ee77e: Status 404 returned error can't find the container with id 64de82a1c9058a4de7ddaec401e15ef29d85131b8f49d1b77adaec11966ee77e
Nov 01 12:51:58.312157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.312116    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"]
Nov 01 12:51:58.318265 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:51:58.318228    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca93cb82_56a8_4a3f_806d_674a21c9d346.slice/crio-9cd55cedbb9cea6e0ff04aced0e0457d5a4fddc48eed5f471e871c30618bd0f5 WatchSource:0}: Error finding container 9cd55cedbb9cea6e0ff04aced0e0457d5a4fddc48eed5f471e871c30618bd0f5: Status 404 returned error can't find the container with id 9cd55cedbb9cea6e0ff04aced0e0457d5a4fddc48eed5f471e871c30618bd0f5
Nov 01 12:51:58.620402 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.620319    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg" event={"ID":"57a2e157-c261-483a-b537-48292fbb16e5","Type":"ContainerStarted","Data":"57de5166a640d1d8ecfaaac34f3057b6358ff96f7d2979ebc63efb0bc7adf479"}
Nov 01 12:51:58.620402 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.620381    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:51:58.620601 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.620406    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg" event={"ID":"57a2e157-c261-483a-b537-48292fbb16e5","Type":"ContainerStarted","Data":"64de82a1c9058a4de7ddaec401e15ef29d85131b8f49d1b77adaec11966ee77e"}
Nov 01 12:51:58.622095 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.622037    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b" event={"ID":"ca93cb82-56a8-4a3f-806d-674a21c9d346","Type":"ContainerStarted","Data":"8e0079937668b7b082b33f8949d9d77df036f84d50c5de7877d99f26473fecdc"}
Nov 01 12:51:58.622205 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.622116    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b" event={"ID":"ca93cb82-56a8-4a3f-806d-674a21c9d346","Type":"ContainerStarted","Data":"9cd55cedbb9cea6e0ff04aced0e0457d5a4fddc48eed5f471e871c30618bd0f5"}
Nov 01 12:51:58.622205 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.622144    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:51:58.729382 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.729327    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg" podStartSLOduration=1.729313906 podStartE2EDuration="1.729313906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:51:58.727782694 +0000 UTC m=+1760.234973912" watchObservedRunningTime="2026-11-01 12:51:58.729313906 +0000 UTC m=+1760.236505122"
Nov 01 12:51:58.842915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:51:58.840074    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b" podStartSLOduration=1.840039652 podStartE2EDuration="1.840039652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:51:58.836179881 +0000 UTC m=+1760.343371098" watchObservedRunningTime="2026-11-01 12:51:58.840039652 +0000 UTC m=+1760.347230870"
Nov 01 12:52:01.028238 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:01.028203    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:52:02.224196 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.224115    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"]
Nov 01 12:52:02.228489 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.228462    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:02.280249 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.280206    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"]
Nov 01 12:52:02.341073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.328496    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c4bbec-dockercfg-7dbhh\""
Nov 01 12:52:02.341073 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.328592    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c4bbec\""
Nov 01 12:52:02.382080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.380957    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2vf\" (UniqueName: \"kubernetes.io/projected/550d565d-e086-4087-9777-9ababfbbd3e2-kube-api-access-nw2vf\") pod \"payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr\" (UID: \"550d565d-e086-4087-9777-9ababfbbd3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:02.382080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.381012    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550d565d-e086-4087-9777-9ababfbbd3e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr\" (UID: \"550d565d-e086-4087-9777-9ababfbbd3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:02.392129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.387105    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"]
Nov 01 12:52:02.392648 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.392626    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:02.433456 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.433424    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"]
Nov 01 12:52:02.460447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.457372    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:52:02.460447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.458510    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:52:02.486170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.483664    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6\" (UID: \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:02.486170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.484839    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2vf\" (UniqueName: \"kubernetes.io/projected/550d565d-e086-4087-9777-9ababfbbd3e2-kube-api-access-nw2vf\") pod \"payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr\" (UID: \"550d565d-e086-4087-9777-9ababfbbd3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:02.486170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.485848    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550d565d-e086-4087-9777-9ababfbbd3e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr\" (UID: \"550d565d-e086-4087-9777-9ababfbbd3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:02.494475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.486799    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqs62\" (UniqueName: \"kubernetes.io/projected/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-kube-api-access-pqs62\") pod \"payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6\" (UID: \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:02.513089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.504755    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550d565d-e086-4087-9777-9ababfbbd3e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr\" (UID: \"550d565d-e086-4087-9777-9ababfbbd3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:02.550169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.550132    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2vf\" (UniqueName: \"kubernetes.io/projected/550d565d-e086-4087-9777-9ababfbbd3e2-kube-api-access-nw2vf\") pod \"payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr\" (UID: \"550d565d-e086-4087-9777-9ababfbbd3e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:02.553729 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.553691    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:02.588308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.588275    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqs62\" (UniqueName: \"kubernetes.io/projected/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-kube-api-access-pqs62\") pod \"payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6\" (UID: \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:02.588469 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.588444    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6\" (UID: \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:02.589515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.589492    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6\" (UID: \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:02.658343 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.657771    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqs62\" (UniqueName: \"kubernetes.io/projected/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-kube-api-access-pqs62\") pod \"payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6\" (UID: \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:02.741622 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.741520    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:02.953738 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:02.953698    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"]
Nov 01 12:52:02.959831 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:52:02.959799    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550d565d_e086_4087_9777_9ababfbbd3e2.slice/crio-c2540ebf55fa934720a7297ab6b82120ce159f00c1cbe916aefa1bef413a22d9 WatchSource:0}: Error finding container c2540ebf55fa934720a7297ab6b82120ce159f00c1cbe916aefa1bef413a22d9: Status 404 returned error can't find the container with id c2540ebf55fa934720a7297ab6b82120ce159f00c1cbe916aefa1bef413a22d9
Nov 01 12:52:03.148971 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:03.148933    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"]
Nov 01 12:52:03.154641 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:52:03.154610    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b9baf9_fd4a_4f71_97d1_6d2b452a55fc.slice/crio-cf9483325cee643800f025a03003cb69144f9185f41f1a8329d9bb99479fc940 WatchSource:0}: Error finding container cf9483325cee643800f025a03003cb69144f9185f41f1a8329d9bb99479fc940: Status 404 returned error can't find the container with id cf9483325cee643800f025a03003cb69144f9185f41f1a8329d9bb99479fc940
Nov 01 12:52:03.677384 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:03.677342    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6" event={"ID":"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc","Type":"ContainerStarted","Data":"41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d"}
Nov 01 12:52:03.677890 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:03.677434    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6" event={"ID":"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc","Type":"ContainerStarted","Data":"cf9483325cee643800f025a03003cb69144f9185f41f1a8329d9bb99479fc940"}
Nov 01 12:52:03.677890 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:03.677538    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:03.679234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:03.679200    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr" event={"ID":"550d565d-e086-4087-9777-9ababfbbd3e2","Type":"ContainerStarted","Data":"ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b"}
Nov 01 12:52:03.679376 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:03.679253    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr" event={"ID":"550d565d-e086-4087-9777-9ababfbbd3e2","Type":"ContainerStarted","Data":"c2540ebf55fa934720a7297ab6b82120ce159f00c1cbe916aefa1bef413a22d9"}
Nov 01 12:52:03.679376 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:03.679279    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:03.789966 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:03.789901    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6" podStartSLOduration=1.789881166 podStartE2EDuration="1.789881166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:52:03.787760604 +0000 UTC m=+1765.294951804" watchObservedRunningTime="2026-11-01 12:52:03.789881166 +0000 UTC m=+1765.297072384"
Nov 01 12:52:03.898919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:03.898865    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr" podStartSLOduration=1.898845828 podStartE2EDuration="1.898845828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:52:03.896119644 +0000 UTC m=+1765.403310864" watchObservedRunningTime="2026-11-01 12:52:03.898845828 +0000 UTC m=+1765.406037046"
Nov 01 12:52:09.643339 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:09.643302    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:52:09.660309 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:09.660275    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:52:14.690321 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:14.690292    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:52:14.690813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:14.690369    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:52:20.845127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:20.845016    5725 generic.go:356] "Generic (PLEG): container finished" podID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerID="91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff" exitCode=0
Nov 01 12:52:20.845127 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:20.845084    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" event={"ID":"385cf49d-a997-42c7-9d2f-52c6dacdd881","Type":"ContainerDied","Data":"91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff"}
Nov 01 12:52:20.845685 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:20.845645    5725 scope.go:122] "RemoveContainer" containerID="91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff"
Nov 01 12:52:20.847705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:20.847669    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:52:20.847900 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:20.847755    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:52:20.847900 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:20.847789    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:52:21.055253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:21.055216    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"]
Nov 01 12:52:21.061447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:21.061405    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034e29cb-wx5s5"]
Nov 01 12:52:21.866891 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:21.866854    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" event={"ID":"385cf49d-a997-42c7-9d2f-52c6dacdd881","Type":"ContainerStarted","Data":"88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7"}
Nov 01 12:52:22.617976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:22.617936    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6192132-20ed-4383-ba12-654628203624" path="/var/lib/kubelet/pods/d6192132-20ed-4383-ba12-654628203624/volumes"
Nov 01 12:52:22.672259 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:22.672220    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:52:22.673438 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:22.673412    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 12:52:22.673548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:22.673469    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 12:52:23.673451 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:23.673408    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 12:52:23.673959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:23.673461    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 12:52:24.672972 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:24.672937    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 12:52:24.673182 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:24.673000    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 12:52:25.673393 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:25.673353    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 12:52:25.673796 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:25.673412    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 12:52:26.672597 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:26.672563    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 12:52:26.672789 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:26.672616    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 12:52:26.989431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:26.989394    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:52:27.677369 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:27.677339    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:52:27.925364 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:27.925336    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 12:52:31.030241 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:31.030203    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:52:33.623339 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:33.623298    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7085-d78f1e"]
Nov 01 12:52:33.627990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:33.627957    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-d78f1e"
Nov 01 12:52:33.676860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:33.676824    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7085-d78f1e"]
Nov 01 12:52:33.713771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:33.713730    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22nhz\" (UniqueName: \"kubernetes.io/projected/e69c89db-91db-4a8e-8ed0-03330ac78f24-kube-api-access-22nhz\") pod \"test-curl-7085-d78f1e\" (UID: \"e69c89db-91db-4a8e-8ed0-03330ac78f24\") " pod="opendatahub/test-curl-7085-d78f1e"
Nov 01 12:52:33.718323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:33.718302    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rbvvl\""
Nov 01 12:52:33.815663 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:33.815628    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22nhz\" (UniqueName: \"kubernetes.io/projected/e69c89db-91db-4a8e-8ed0-03330ac78f24-kube-api-access-22nhz\") pod \"test-curl-7085-d78f1e\" (UID: \"e69c89db-91db-4a8e-8ed0-03330ac78f24\") " pod="opendatahub/test-curl-7085-d78f1e"
Nov 01 12:52:33.870286 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:33.870253    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22nhz\" (UniqueName: \"kubernetes.io/projected/e69c89db-91db-4a8e-8ed0-03330ac78f24-kube-api-access-22nhz\") pod \"test-curl-7085-d78f1e\" (UID: \"e69c89db-91db-4a8e-8ed0-03330ac78f24\") " pod="opendatahub/test-curl-7085-d78f1e"
Nov 01 12:52:33.941438 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:33.941359    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-d78f1e"
Nov 01 12:52:34.317392 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:34.317357    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7085-d78f1e"]
Nov 01 12:52:34.323583 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:52:34.323558    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode69c89db_91db_4a8e_8ed0_03330ac78f24.slice/crio-a26fc1e7a6e5eb43ddf5e1346f22f7859c2e29a87073d383e7a70142e33c73b5 WatchSource:0}: Error finding container a26fc1e7a6e5eb43ddf5e1346f22f7859c2e29a87073d383e7a70142e33c73b5: Status 404 returned error can't find the container with id a26fc1e7a6e5eb43ddf5e1346f22f7859c2e29a87073d383e7a70142e33c73b5
Nov 01 12:52:34.984364 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:34.984323    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7085-d78f1e" event={"ID":"e69c89db-91db-4a8e-8ed0-03330ac78f24","Type":"ContainerStarted","Data":"145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2"}
Nov 01 12:52:34.984744 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:34.984373    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7085-d78f1e" event={"ID":"e69c89db-91db-4a8e-8ed0-03330ac78f24","Type":"ContainerStarted","Data":"a26fc1e7a6e5eb43ddf5e1346f22f7859c2e29a87073d383e7a70142e33c73b5"}
Nov 01 12:52:35.093211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:35.093145    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7085-d78f1e" podStartSLOduration=2.093126221 podStartE2EDuration="2.093126221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:52:35.090347741 +0000 UTC m=+1796.597538970" watchObservedRunningTime="2026-11-01 12:52:35.093126221 +0000 UTC m=+1796.600317439"
Nov 01 12:52:36.492934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.492896    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7085-d78f1e"]
Nov 01 12:52:36.497409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.497382    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7085-d78f1e"]
Nov 01 12:52:36.633043 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.633000    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7085-80ff4e"]
Nov 01 12:52:36.633721 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.633701    5725 cpu_manager.go:414] "containerMap: removing container" podUID="e69c89db-91db-4a8e-8ed0-03330ac78f24" containerName="test-curl-7085-d78f1e"
Nov 01 12:52:36.633721 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.633721    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69c89db-91db-4a8e-8ed0-03330ac78f24" containerName="test-curl-7085-d78f1e"
Nov 01 12:52:36.633985 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.633968    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69c89db-91db-4a8e-8ed0-03330ac78f24" containerName="test-curl-7085-d78f1e"
Nov 01 12:52:36.637547 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.637527    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-80ff4e"
Nov 01 12:52:36.688187 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.688124    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7085-80ff4e"]
Nov 01 12:52:36.765576 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.765496    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv5wt\" (UniqueName: \"kubernetes.io/projected/50cf9396-54d2-45c1-8666-586e1da1742b-kube-api-access-wv5wt\") pod \"test-curl-7085-80ff4e\" (UID: \"50cf9396-54d2-45c1-8666-586e1da1742b\") " pod="opendatahub/test-curl-7085-80ff4e"
Nov 01 12:52:36.866696 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.866660    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv5wt\" (UniqueName: \"kubernetes.io/projected/50cf9396-54d2-45c1-8666-586e1da1742b-kube-api-access-wv5wt\") pod \"test-curl-7085-80ff4e\" (UID: \"50cf9396-54d2-45c1-8666-586e1da1742b\") " pod="opendatahub/test-curl-7085-80ff4e"
Nov 01 12:52:36.920655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.920621    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv5wt\" (UniqueName: \"kubernetes.io/projected/50cf9396-54d2-45c1-8666-586e1da1742b-kube-api-access-wv5wt\") pod \"test-curl-7085-80ff4e\" (UID: \"50cf9396-54d2-45c1-8666-586e1da1742b\") " pod="opendatahub/test-curl-7085-80ff4e"
Nov 01 12:52:36.953766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:36.953734    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-80ff4e"
Nov 01 12:52:37.003688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:37.003650    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7085-d78f1e" podUID="e69c89db-91db-4a8e-8ed0-03330ac78f24" containerName="test-curl-7085-d78f1e" containerID="cri-o://145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2" gracePeriod=2
Nov 01 12:52:37.398958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:37.398875    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7085-80ff4e"]
Nov 01 12:52:37.404926 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:52:37.404894    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50cf9396_54d2_45c1_8666_586e1da1742b.slice/crio-9beaef2961c4d45184d796bcf903fb0104598dfd06aa7000186d280092b35dcc WatchSource:0}: Error finding container 9beaef2961c4d45184d796bcf903fb0104598dfd06aa7000186d280092b35dcc: Status 404 returned error can't find the container with id 9beaef2961c4d45184d796bcf903fb0104598dfd06aa7000186d280092b35dcc
Nov 01 12:52:38.012850 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:38.012803    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7085-80ff4e" event={"ID":"50cf9396-54d2-45c1-8666-586e1da1742b","Type":"ContainerStarted","Data":"4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0"}
Nov 01 12:52:38.012850 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:38.012848    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7085-80ff4e" event={"ID":"50cf9396-54d2-45c1-8666-586e1da1742b","Type":"ContainerStarted","Data":"9beaef2961c4d45184d796bcf903fb0104598dfd06aa7000186d280092b35dcc"}
Nov 01 12:52:38.121200 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:38.121149    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7085-80ff4e" podStartSLOduration=2.121133305 podStartE2EDuration="2.121133305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:52:38.118488175 +0000 UTC m=+1799.625679374" watchObservedRunningTime="2026-11-01 12:52:38.121133305 +0000 UTC m=+1799.628324522"
Nov 01 12:52:39.040830 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.040555    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:52:39.050291 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.050266    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:52:39.051397 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.051377    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:52:39.061590 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.061569    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:52:39.264996 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.264929    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-d78f1e"
Nov 01 12:52:39.401614 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.401581    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69c89db-91db-4a8e-8ed0-03330ac78f24-kube-api-access-22nhz\" (UniqueName: \"kubernetes.io/projected/e69c89db-91db-4a8e-8ed0-03330ac78f24-kube-api-access-22nhz\") pod \"e69c89db-91db-4a8e-8ed0-03330ac78f24\" (UID: \"e69c89db-91db-4a8e-8ed0-03330ac78f24\") "
Nov 01 12:52:39.403749 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.403722    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69c89db-91db-4a8e-8ed0-03330ac78f24-kube-api-access-22nhz" pod "e69c89db-91db-4a8e-8ed0-03330ac78f24" (UID: "e69c89db-91db-4a8e-8ed0-03330ac78f24"). InnerVolumeSpecName "kube-api-access-22nhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:52:39.503625 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.503593    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22nhz\" (UniqueName: \"kubernetes.io/projected/e69c89db-91db-4a8e-8ed0-03330ac78f24-kube-api-access-22nhz\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:52:39.537876 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.537797    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7085-80ff4e"]
Nov 01 12:52:39.543766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.543725    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7085-80ff4e"]
Nov 01 12:52:39.685310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.685272    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7085-49ea5b"]
Nov 01 12:52:39.686280 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.686260    5725 cpu_manager.go:414] "containerMap: removing container" podUID="50cf9396-54d2-45c1-8666-586e1da1742b" containerName="test-curl-7085-80ff4e"
Nov 01 12:52:39.686339 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.686283    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cf9396-54d2-45c1-8666-586e1da1742b" containerName="test-curl-7085-80ff4e"
Nov 01 12:52:39.686477 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.686466    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cf9396-54d2-45c1-8666-586e1da1742b" containerName="test-curl-7085-80ff4e"
Nov 01 12:52:39.690508 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.690483    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-49ea5b"
Nov 01 12:52:39.740442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.740409    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7085-49ea5b"]
Nov 01 12:52:39.808199 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.808109    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zpb\" (UniqueName: \"kubernetes.io/projected/655a5d95-66c6-45fd-a3e0-29a1c7dcd846-kube-api-access-x9zpb\") pod \"test-curl-7085-49ea5b\" (UID: \"655a5d95-66c6-45fd-a3e0-29a1c7dcd846\") " pod="opendatahub/test-curl-7085-49ea5b"
Nov 01 12:52:39.909169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.909129    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zpb\" (UniqueName: \"kubernetes.io/projected/655a5d95-66c6-45fd-a3e0-29a1c7dcd846-kube-api-access-x9zpb\") pod \"test-curl-7085-49ea5b\" (UID: \"655a5d95-66c6-45fd-a3e0-29a1c7dcd846\") " pod="opendatahub/test-curl-7085-49ea5b"
Nov 01 12:52:39.964560 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:39.964522    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zpb\" (UniqueName: \"kubernetes.io/projected/655a5d95-66c6-45fd-a3e0-29a1c7dcd846-kube-api-access-x9zpb\") pod \"test-curl-7085-49ea5b\" (UID: \"655a5d95-66c6-45fd-a3e0-29a1c7dcd846\") " pod="opendatahub/test-curl-7085-49ea5b"
Nov 01 12:52:40.004915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:40.004878    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-49ea5b"
Nov 01 12:52:40.045082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:40.036277    5725 generic.go:356] "Generic (PLEG): container finished" podID="e69c89db-91db-4a8e-8ed0-03330ac78f24" containerID="145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2" exitCode=137
Nov 01 12:52:40.045082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:40.036542    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7085-80ff4e" podUID="50cf9396-54d2-45c1-8666-586e1da1742b" containerName="test-curl-7085-80ff4e" containerID="cri-o://4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0" gracePeriod=2
Nov 01 12:52:40.045082 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:40.036703    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-d78f1e"
Nov 01 12:52:40.070539 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:40.046889    5725 scope.go:122] "RemoveContainer" containerID="145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2"
Nov 01 12:52:40.072220 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:40.072183    5725 scope.go:122] "RemoveContainer" containerID="145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2"
Nov 01 12:52:40.072641 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:40.072613    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2\": container with ID starting with 145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2 not found: ID does not exist" containerID="145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2"
Nov 01 12:52:40.072743 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:40.072655    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2"} err="failed to get container status \"145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2\": rpc error: code = NotFound desc = could not find container \"145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2\": container with ID starting with 145812e75446d52ad070fd76c1774691bb28e662d8d0f0ea25bb4a196033ebb2 not found: ID does not exist"
Nov 01 12:52:40.096368 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:40.096215    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7085-80ff4e\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="50cf9396-54d2-45c1-8666-586e1da1742b" pod="opendatahub/test-curl-7085-80ff4e"
Nov 01 12:52:40.146027 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:40.144494    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7085-d78f1e\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="e69c89db-91db-4a8e-8ed0-03330ac78f24" pod="opendatahub/test-curl-7085-d78f1e"
Nov 01 12:52:40.192508 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:40.192469    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7085-80ff4e\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="50cf9396-54d2-45c1-8666-586e1da1742b" pod="opendatahub/test-curl-7085-80ff4e"
Nov 01 12:52:40.410616 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:40.410516    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7085-49ea5b"]
Nov 01 12:52:40.415550 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:52:40.415515    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod655a5d95_66c6_45fd_a3e0_29a1c7dcd846.slice/crio-553609a3794f998198d5bac7106029d55741358367fb2094aeaf1882e86bdd2c WatchSource:0}: Error finding container 553609a3794f998198d5bac7106029d55741358367fb2094aeaf1882e86bdd2c: Status 404 returned error can't find the container with id 553609a3794f998198d5bac7106029d55741358367fb2094aeaf1882e86bdd2c
Nov 01 12:52:40.621916 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:40.621878    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69c89db-91db-4a8e-8ed0-03330ac78f24" path="/var/lib/kubelet/pods/e69c89db-91db-4a8e-8ed0-03330ac78f24/volumes"
Nov 01 12:52:41.051068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:41.050363    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7085-49ea5b" event={"ID":"655a5d95-66c6-45fd-a3e0-29a1c7dcd846","Type":"ContainerStarted","Data":"f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155"}
Nov 01 12:52:41.051068 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:41.050411    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7085-49ea5b" event={"ID":"655a5d95-66c6-45fd-a3e0-29a1c7dcd846","Type":"ContainerStarted","Data":"553609a3794f998198d5bac7106029d55741358367fb2094aeaf1882e86bdd2c"}
Nov 01 12:52:41.159700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:41.159650    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7085-49ea5b" podStartSLOduration=2.159633718 podStartE2EDuration="2.159633718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:52:41.156641538 +0000 UTC m=+1802.663832783" watchObservedRunningTime="2026-11-01 12:52:41.159633718 +0000 UTC m=+1802.666824935"
Nov 01 12:52:42.318029 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.317969    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-80ff4e"
Nov 01 12:52:42.444522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.444485    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cf9396-54d2-45c1-8666-586e1da1742b-kube-api-access-wv5wt\" (UniqueName: \"kubernetes.io/projected/50cf9396-54d2-45c1-8666-586e1da1742b-kube-api-access-wv5wt\") pod \"50cf9396-54d2-45c1-8666-586e1da1742b\" (UID: \"50cf9396-54d2-45c1-8666-586e1da1742b\") "
Nov 01 12:52:42.446740 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.446705    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cf9396-54d2-45c1-8666-586e1da1742b-kube-api-access-wv5wt" pod "50cf9396-54d2-45c1-8666-586e1da1742b" (UID: "50cf9396-54d2-45c1-8666-586e1da1742b"). InnerVolumeSpecName "kube-api-access-wv5wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:52:42.457192 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.457163    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7085-49ea5b"]
Nov 01 12:52:42.469861 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.464335    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7085-49ea5b"]
Nov 01 12:52:42.546900 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.546860    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv5wt\" (UniqueName: \"kubernetes.io/projected/50cf9396-54d2-45c1-8666-586e1da1742b-kube-api-access-wv5wt\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:52:42.597996 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.597914    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7085-a014c0"]
Nov 01 12:52:42.598597 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.598581    5725 cpu_manager.go:414] "containerMap: removing container" podUID="655a5d95-66c6-45fd-a3e0-29a1c7dcd846" containerName="test-curl-7085-49ea5b"
Nov 01 12:52:42.598653 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.598598    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655a5d95-66c6-45fd-a3e0-29a1c7dcd846" containerName="test-curl-7085-49ea5b"
Nov 01 12:52:42.598714 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.598705    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="655a5d95-66c6-45fd-a3e0-29a1c7dcd846" containerName="test-curl-7085-49ea5b"
Nov 01 12:52:42.602346 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.602325    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-a014c0"
Nov 01 12:52:42.618163 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.618136    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cf9396-54d2-45c1-8666-586e1da1742b" path="/var/lib/kubelet/pods/50cf9396-54d2-45c1-8666-586e1da1742b/volumes"
Nov 01 12:52:42.654633 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.654594    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7085-a014c0"]
Nov 01 12:52:42.750001 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.749962    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sbvh\" (UniqueName: \"kubernetes.io/projected/16628597-99fb-4a21-bb60-939efd77e12d-kube-api-access-2sbvh\") pod \"test-curl-7085-a014c0\" (UID: \"16628597-99fb-4a21-bb60-939efd77e12d\") " pod="opendatahub/test-curl-7085-a014c0"
Nov 01 12:52:42.852407 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.852313    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sbvh\" (UniqueName: \"kubernetes.io/projected/16628597-99fb-4a21-bb60-939efd77e12d-kube-api-access-2sbvh\") pod \"test-curl-7085-a014c0\" (UID: \"16628597-99fb-4a21-bb60-939efd77e12d\") " pod="opendatahub/test-curl-7085-a014c0"
Nov 01 12:52:42.906870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.906836    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sbvh\" (UniqueName: \"kubernetes.io/projected/16628597-99fb-4a21-bb60-939efd77e12d-kube-api-access-2sbvh\") pod \"test-curl-7085-a014c0\" (UID: \"16628597-99fb-4a21-bb60-939efd77e12d\") " pod="opendatahub/test-curl-7085-a014c0"
Nov 01 12:52:42.915785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:42.915757    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-a014c0"
Nov 01 12:52:43.074743 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:43.074696    5725 generic.go:356] "Generic (PLEG): container finished" podID="50cf9396-54d2-45c1-8666-586e1da1742b" containerID="4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0" exitCode=137
Nov 01 12:52:43.074932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:43.074837    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-80ff4e"
Nov 01 12:52:43.075069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:43.074999    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7085-49ea5b" podUID="655a5d95-66c6-45fd-a3e0-29a1c7dcd846" containerName="test-curl-7085-49ea5b" containerID="cri-o://f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155" gracePeriod=2
Nov 01 12:52:43.075190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:43.075171    5725 scope.go:122] "RemoveContainer" containerID="4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0"
Nov 01 12:52:43.086898 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:43.086874    5725 scope.go:122] "RemoveContainer" containerID="4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0"
Nov 01 12:52:43.087231 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:43.087212    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0\": container with ID starting with 4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0 not found: ID does not exist" containerID="4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0"
Nov 01 12:52:43.087280 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:43.087240    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0"} err="failed to get container status \"4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0\": rpc error: code = NotFound desc = could not find container \"4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0\": container with ID starting with 4428b1424c790a8187d1ef708e97a0ac28e65f936cd80664c1cc7b4c8cf837c0 not found: ID does not exist"
Nov 01 12:52:43.306649 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:43.306614    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7085-a014c0"]
Nov 01 12:52:43.312164 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:52:43.312131    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16628597_99fb_4a21_bb60_939efd77e12d.slice/crio-8cd5aa721a3304eac68432f6f4cd97071f1a5fa150348a39e468219f50834857 WatchSource:0}: Error finding container 8cd5aa721a3304eac68432f6f4cd97071f1a5fa150348a39e468219f50834857: Status 404 returned error can't find the container with id 8cd5aa721a3304eac68432f6f4cd97071f1a5fa150348a39e468219f50834857
Nov 01 12:52:44.084493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:44.084453    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7085-a014c0" event={"ID":"16628597-99fb-4a21-bb60-939efd77e12d","Type":"ContainerStarted","Data":"6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9"}
Nov 01 12:52:44.084493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:44.084503    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7085-a014c0" event={"ID":"16628597-99fb-4a21-bb60-939efd77e12d","Type":"ContainerStarted","Data":"8cd5aa721a3304eac68432f6f4cd97071f1a5fa150348a39e468219f50834857"}
Nov 01 12:52:44.196766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:44.196711    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7085-a014c0" podStartSLOduration=2.196698006 podStartE2EDuration="2.196698006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:52:44.193946553 +0000 UTC m=+1805.701137770" watchObservedRunningTime="2026-11-01 12:52:44.196698006 +0000 UTC m=+1805.703889222"
Nov 01 12:52:45.347142 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:45.347081    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-49ea5b"
Nov 01 12:52:45.488958 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:45.488928    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655a5d95-66c6-45fd-a3e0-29a1c7dcd846-kube-api-access-x9zpb\" (UniqueName: \"kubernetes.io/projected/655a5d95-66c6-45fd-a3e0-29a1c7dcd846-kube-api-access-x9zpb\") pod \"655a5d95-66c6-45fd-a3e0-29a1c7dcd846\" (UID: \"655a5d95-66c6-45fd-a3e0-29a1c7dcd846\") "
Nov 01 12:52:45.490959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:45.490928    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655a5d95-66c6-45fd-a3e0-29a1c7dcd846-kube-api-access-x9zpb" pod "655a5d95-66c6-45fd-a3e0-29a1c7dcd846" (UID: "655a5d95-66c6-45fd-a3e0-29a1c7dcd846"). InnerVolumeSpecName "kube-api-access-x9zpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:52:45.590469 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:45.590434    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9zpb\" (UniqueName: \"kubernetes.io/projected/655a5d95-66c6-45fd-a3e0-29a1c7dcd846-kube-api-access-x9zpb\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:52:45.640296 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:45.640223    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7085-a014c0"]
Nov 01 12:52:45.645183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:45.645151    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7085-a014c0"]
Nov 01 12:52:46.102890 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:46.102847    5725 generic.go:356] "Generic (PLEG): container finished" podID="655a5d95-66c6-45fd-a3e0-29a1c7dcd846" containerID="f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155" exitCode=137
Nov 01 12:52:46.102890 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:46.102894    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-49ea5b"
Nov 01 12:52:46.103157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:46.102948    5725 scope.go:122] "RemoveContainer" containerID="f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155"
Nov 01 12:52:46.103157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:46.103103    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7085-a014c0" podUID="16628597-99fb-4a21-bb60-939efd77e12d" containerName="test-curl-7085-a014c0" containerID="cri-o://6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9" gracePeriod=2
Nov 01 12:52:46.117111 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:46.117088    5725 scope.go:122] "RemoveContainer" containerID="f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155"
Nov 01 12:52:46.117434 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:46.117402    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155\": container with ID starting with f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155 not found: ID does not exist" containerID="f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155"
Nov 01 12:52:46.117561 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:46.117439    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155"} err="failed to get container status \"f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155\": rpc error: code = NotFound desc = could not find container \"f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155\": container with ID starting with f93a1dd1fd311719688cb4d601c91e3d081f49f45f143c5b2b9be0e500524155 not found: ID does not exist"
Nov 01 12:52:46.151585 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:46.151547    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7085-a014c0\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="16628597-99fb-4a21-bb60-939efd77e12d" pod="opendatahub/test-curl-7085-a014c0"
Nov 01 12:52:46.199787 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:46.199735    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7085-49ea5b\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="655a5d95-66c6-45fd-a3e0-29a1c7dcd846" pod="opendatahub/test-curl-7085-49ea5b"
Nov 01 12:52:46.247663 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:46.247612    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7085-a014c0\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="16628597-99fb-4a21-bb60-939efd77e12d" pod="opendatahub/test-curl-7085-a014c0"
Nov 01 12:52:46.619212 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:46.619164    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655a5d95-66c6-45fd-a3e0-29a1c7dcd846" path="/var/lib/kubelet/pods/655a5d95-66c6-45fd-a3e0-29a1c7dcd846/volumes"
Nov 01 12:52:48.391896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:48.391868    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-a014c0"
Nov 01 12:52:48.449108 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:48.449077    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16628597-99fb-4a21-bb60-939efd77e12d-kube-api-access-2sbvh\" (UniqueName: \"kubernetes.io/projected/16628597-99fb-4a21-bb60-939efd77e12d-kube-api-access-2sbvh\") pod \"16628597-99fb-4a21-bb60-939efd77e12d\" (UID: \"16628597-99fb-4a21-bb60-939efd77e12d\") "
Nov 01 12:52:48.451269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:48.451237    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16628597-99fb-4a21-bb60-939efd77e12d-kube-api-access-2sbvh" pod "16628597-99fb-4a21-bb60-939efd77e12d" (UID: "16628597-99fb-4a21-bb60-939efd77e12d"). InnerVolumeSpecName "kube-api-access-2sbvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:52:48.550822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:48.550785    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sbvh\" (UniqueName: \"kubernetes.io/projected/16628597-99fb-4a21-bb60-939efd77e12d-kube-api-access-2sbvh\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:52:48.623859 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:48.623768    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16628597-99fb-4a21-bb60-939efd77e12d" path="/var/lib/kubelet/pods/16628597-99fb-4a21-bb60-939efd77e12d/volumes"
Nov 01 12:52:49.150173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:49.150130    5725 generic.go:356] "Generic (PLEG): container finished" podID="16628597-99fb-4a21-bb60-939efd77e12d" containerID="6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9" exitCode=137
Nov 01 12:52:49.150173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:49.150178    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7085-a014c0"
Nov 01 12:52:49.150453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:49.150231    5725 scope.go:122] "RemoveContainer" containerID="6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9"
Nov 01 12:52:49.160909 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:49.160885    5725 scope.go:122] "RemoveContainer" containerID="6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9"
Nov 01 12:52:49.161220 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:49.161199    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9\": container with ID starting with 6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9 not found: ID does not exist" containerID="6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9"
Nov 01 12:52:49.161292 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:49.161230    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9"} err="failed to get container status \"6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9\": rpc error: code = NotFound desc = could not find container \"6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9\": container with ID starting with 6076a242d930d2cd5568676b52fd78ce80b065691b9943b33d5b298e738663a9 not found: ID does not exist"
Nov 01 12:52:58.101700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.101666    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"]
Nov 01 12:52:58.102568 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.102546    5725 cpu_manager.go:414] "containerMap: removing container" podUID="16628597-99fb-4a21-bb60-939efd77e12d" containerName="test-curl-7085-a014c0"
Nov 01 12:52:58.102619 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.102570    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16628597-99fb-4a21-bb60-939efd77e12d" containerName="test-curl-7085-a014c0"
Nov 01 12:52:58.102706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.102697    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="16628597-99fb-4a21-bb60-939efd77e12d" containerName="test-curl-7085-a014c0"
Nov 01 12:52:58.107590 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.107570    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.161780 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.161748    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"]
Nov 01 12:52:58.192326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.192291    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdxn\" (UniqueName: \"kubernetes.io/projected/1a8dbd25-9551-4b82-931e-dfda55ba2214-kube-api-access-ssdxn\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.192591 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.192562    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dbd25-9551-4b82-931e-dfda55ba2214-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.192728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.192668    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.192797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.192775    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.203211 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.203182    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-6124f9-serving-cert\""
Nov 01 12:52:58.380623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.380459    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdxn\" (UniqueName: \"kubernetes.io/projected/1a8dbd25-9551-4b82-931e-dfda55ba2214-kube-api-access-ssdxn\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.380623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.380526    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dbd25-9551-4b82-931e-dfda55ba2214-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.380623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.380593    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.380846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.380701    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.382227 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:58.382171    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-serving-cert: secret "maas-api-e2e-shared-a-w4-6124f9-serving-cert" not found
Nov 01 12:52:58.382364 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:52:58.382264    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls podName:1a8dbd25-9551-4b82-931e-dfda55ba2214 nodeName:}" failed. No retries permitted until 2026-11-01 12:52:58.882243578 +0000 UTC m=+1820.389434797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls") pod "maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q" (UID: "1a8dbd25-9551-4b82-931e-dfda55ba2214") : secret "maas-api-e2e-shared-a-w4-6124f9-serving-cert" not found
Nov 01 12:52:58.382609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.382584    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dbd25-9551-4b82-931e-dfda55ba2214-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.399092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.397741    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.436932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.436893    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdxn\" (UniqueName: \"kubernetes.io/projected/1a8dbd25-9551-4b82-931e-dfda55ba2214-kube-api-access-ssdxn\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.540747 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.540710    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"]
Nov 01 12:52:58.545921 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.545895    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:52:58.590191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.590156    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33626a56-ce7d-49b8-9f98-316044f8fa80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg\" (UID: \"33626a56-ce7d-49b8-9f98-316044f8fa80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:52:58.590409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.590387    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vml\" (UniqueName: \"kubernetes.io/projected/33626a56-ce7d-49b8-9f98-316044f8fa80-kube-api-access-j9vml\") pod \"payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg\" (UID: \"33626a56-ce7d-49b8-9f98-316044f8fa80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:52:58.597183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.597154    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f893c2\""
Nov 01 12:52:58.598152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.598132    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f893c2-dockercfg-wjr2b\""
Nov 01 12:52:58.600905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.600877    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"]
Nov 01 12:52:58.693119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.693017    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vml\" (UniqueName: \"kubernetes.io/projected/33626a56-ce7d-49b8-9f98-316044f8fa80-kube-api-access-j9vml\") pod \"payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg\" (UID: \"33626a56-ce7d-49b8-9f98-316044f8fa80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:52:58.693287 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.693258    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33626a56-ce7d-49b8-9f98-316044f8fa80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg\" (UID: \"33626a56-ce7d-49b8-9f98-316044f8fa80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:52:58.696566 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.696533    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33626a56-ce7d-49b8-9f98-316044f8fa80-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg\" (UID: \"33626a56-ce7d-49b8-9f98-316044f8fa80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:52:58.696780 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.696659    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"]
Nov 01 12:52:58.735448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.735414    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:52:58.852074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.827413    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc7wb\" (UniqueName: \"kubernetes.io/projected/5b3b02e0-06f8-4865-8301-fc5f33a74972-kube-api-access-kc7wb\") pod \"payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc\" (UID: \"5b3b02e0-06f8-4865-8301-fc5f33a74972\") " pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:52:58.852603 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.852577    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3b02e0-06f8-4865-8301-fc5f33a74972-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc\" (UID: \"5b3b02e0-06f8-4865-8301-fc5f33a74972\") " pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:52:58.853079 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.831342    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vml\" (UniqueName: \"kubernetes.io/projected/33626a56-ce7d-49b8-9f98-316044f8fa80-kube-api-access-j9vml\") pod \"payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg\" (UID: \"33626a56-ce7d-49b8-9f98-316044f8fa80\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:52:58.878074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.864730    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:52:58.878642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.878611    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"]
Nov 01 12:52:58.953852 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.953817    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc7wb\" (UniqueName: \"kubernetes.io/projected/5b3b02e0-06f8-4865-8301-fc5f33a74972-kube-api-access-kc7wb\") pod \"payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc\" (UID: \"5b3b02e0-06f8-4865-8301-fc5f33a74972\") " pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:52:58.954030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.953931    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3b02e0-06f8-4865-8301-fc5f33a74972-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc\" (UID: \"5b3b02e0-06f8-4865-8301-fc5f33a74972\") " pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:52:58.954030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.953980    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:58.955109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.955079    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3b02e0-06f8-4865-8301-fc5f33a74972-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc\" (UID: \"5b3b02e0-06f8-4865-8301-fc5f33a74972\") " pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:52:58.956832 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:58.956811    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:59.017728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:59.017566    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc7wb\" (UniqueName: \"kubernetes.io/projected/5b3b02e0-06f8-4865-8301-fc5f33a74972-kube-api-access-kc7wb\") pod \"payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc\" (UID: \"5b3b02e0-06f8-4865-8301-fc5f33a74972\") " pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:52:59.022325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:59.022029    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:52:59.165653 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:59.165528    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:52:59.330724 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:59.330646    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"]
Nov 01 12:52:59.336715 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:52:59.336685    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33626a56_ce7d_49b8_9f98_316044f8fa80.slice/crio-2968adaa0c57996e63ee4597068751c661c714ae26c831e382b8b666b02c8d67 WatchSource:0}: Error finding container 2968adaa0c57996e63ee4597068751c661c714ae26c831e382b8b666b02c8d67: Status 404 returned error can't find the container with id 2968adaa0c57996e63ee4597068751c661c714ae26c831e382b8b666b02c8d67
Nov 01 12:52:59.401940 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:59.401785    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg" event={"ID":"33626a56-ce7d-49b8-9f98-316044f8fa80","Type":"ContainerStarted","Data":"2968adaa0c57996e63ee4597068751c661c714ae26c831e382b8b666b02c8d67"}
Nov 01 12:52:59.446281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:59.446248    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"]
Nov 01 12:52:59.451754 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:52:59.451713    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8dbd25_9551_4b82_931e_dfda55ba2214.slice/crio-6f1dd31715cfd90cb890681fd4093584ec93156ff584e5d7d635113b96c90b8a WatchSource:0}: Error finding container 6f1dd31715cfd90cb890681fd4093584ec93156ff584e5d7d635113b96c90b8a: Status 404 returned error can't find the container with id 6f1dd31715cfd90cb890681fd4093584ec93156ff584e5d7d635113b96c90b8a
Nov 01 12:52:59.560990 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:52:59.560907    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"]
Nov 01 12:52:59.566797 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:52:59.566762    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3b02e0_06f8_4865_8301_fc5f33a74972.slice/crio-21705c99e8724c3b27a1041df2feb7c9400ef67bd33145626d1a1b63e90cc491 WatchSource:0}: Error finding container 21705c99e8724c3b27a1041df2feb7c9400ef67bd33145626d1a1b63e90cc491: Status 404 returned error can't find the container with id 21705c99e8724c3b27a1041df2feb7c9400ef67bd33145626d1a1b63e90cc491
Nov 01 12:53:00.413500 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:00.413468    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg" event={"ID":"33626a56-ce7d-49b8-9f98-316044f8fa80","Type":"ContainerStarted","Data":"d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a"}
Nov 01 12:53:00.413894 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:00.413579    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:53:00.414965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:00.414940    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q" event={"ID":"1a8dbd25-9551-4b82-931e-dfda55ba2214","Type":"ContainerStarted","Data":"6f1dd31715cfd90cb890681fd4093584ec93156ff584e5d7d635113b96c90b8a"}
Nov 01 12:53:00.416704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:00.416684    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc" event={"ID":"5b3b02e0-06f8-4865-8301-fc5f33a74972","Type":"ContainerStarted","Data":"1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d"}
Nov 01 12:53:00.416790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:00.416719    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc" event={"ID":"5b3b02e0-06f8-4865-8301-fc5f33a74972","Type":"ContainerStarted","Data":"21705c99e8724c3b27a1041df2feb7c9400ef67bd33145626d1a1b63e90cc491"}
Nov 01 12:53:00.416790 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:00.416742    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:53:00.524814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:00.524762    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg" podStartSLOduration=2.524749804 podStartE2EDuration="2.524749804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:00.521224745 +0000 UTC m=+1822.028415961" watchObservedRunningTime="2026-11-01 12:53:00.524749804 +0000 UTC m=+1822.031941020"
Nov 01 12:53:00.634585 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:00.634529    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc" podStartSLOduration=2.634515279 podStartE2EDuration="2.634515279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:00.631567224 +0000 UTC m=+1822.138758465" watchObservedRunningTime="2026-11-01 12:53:00.634515279 +0000 UTC m=+1822.141706496"
Nov 01 12:53:01.036248 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:53:01.036207    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:53:01.400447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.400358    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"]
Nov 01 12:53:01.404808 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.404790    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.430657 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.430616    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q" event={"ID":"1a8dbd25-9551-4b82-931e-dfda55ba2214","Type":"ContainerStarted","Data":"46b7cc0afa25149826245cff728bfd31923e19f34c542fc3afb1b13c56551247"}
Nov 01 12:53:01.431164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.430696    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:53:01.471802 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.471762    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"]
Nov 01 12:53:01.492533 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.492489    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-model-cache\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.492728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.492622    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzmqz\" (UniqueName: \"kubernetes.io/projected/1757ffa6-b479-45f0-bb5f-5d785bb49543-kube-api-access-rzmqz\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.492728 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.492704    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-dshm\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.492819 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.492796    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1757ffa6-b479-45f0-bb5f-5d785bb49543-tls-certs\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.492867 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.492852    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-home\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.492999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.492974    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-tmp-dir\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.499478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.499453    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-187d06ed\"/\"openshift-service-ca.crt\""
Nov 01 12:53:01.499478 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.499461    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-187d06ed\"/\"kube-root-ca.crt\""
Nov 01 12:53:01.499662 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.499509    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-187d06ed\"/\"e2e-lifecycle-model-187d06ed-kserve-self-signed-certs\""
Nov 01 12:53:01.499662 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.499585    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-187d06ed\"/\"default-dockercfg-c2d72\""
Nov 01 12:53:01.594147 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.594098    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-model-cache\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.594325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.594192    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzmqz\" (UniqueName: \"kubernetes.io/projected/1757ffa6-b479-45f0-bb5f-5d785bb49543-kube-api-access-rzmqz\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.594325 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.594303    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-dshm\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.594475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.594361    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1757ffa6-b479-45f0-bb5f-5d785bb49543-tls-certs\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.594475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.594418    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-home\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.594590 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.594489    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-tmp-dir\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.594648 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.594615    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-model-cache\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.594919 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.594893    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-home\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.595209 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.595187    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-tmp-dir\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.596878 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.596857    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-dshm\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.597312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.597292    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1757ffa6-b479-45f0-bb5f-5d785bb49543-tls-certs\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.638069 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.638001    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q" podStartSLOduration=2.702003268 podStartE2EDuration="3.637980225s" totalImagesPullingTime="935.976957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:01.635241183 +0000 UTC m=+1823.142432411" watchObservedRunningTime="2026-11-01 12:53:01.637980225 +0000 UTC m=+1823.145171443"
Nov 01 12:53:01.659579 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.659491    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzmqz\" (UniqueName: \"kubernetes.io/projected/1757ffa6-b479-45f0-bb5f-5d785bb49543-kube-api-access-rzmqz\") pod \"e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") " pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:01.717645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:01.717612    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:02.117982 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:02.117926    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"]
Nov 01 12:53:02.123655 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:53:02.123619    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1757ffa6_b479_45f0_bb5f_5d785bb49543.slice/crio-b0376e80066dd0764b38ee488ed2b112f739dd1d937ca017e849ec0db714634c WatchSource:0}: Error finding container b0376e80066dd0764b38ee488ed2b112f739dd1d937ca017e849ec0db714634c: Status 404 returned error can't find the container with id b0376e80066dd0764b38ee488ed2b112f739dd1d937ca017e849ec0db714634c
Nov 01 12:53:02.443089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:02.442972    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c" event={"ID":"1757ffa6-b479-45f0-bb5f-5d785bb49543","Type":"ContainerStarted","Data":"6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e"}
Nov 01 12:53:02.443089 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:02.443029    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c" event={"ID":"1757ffa6-b479-45f0-bb5f-5d785bb49543","Type":"ContainerStarted","Data":"b0376e80066dd0764b38ee488ed2b112f739dd1d937ca017e849ec0db714634c"}
Nov 01 12:53:02.554938 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:02.554865    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c" podStartSLOduration=1.554846161 podStartE2EDuration="1.554846161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:02.551581565 +0000 UTC m=+1824.058772786" watchObservedRunningTime="2026-11-01 12:53:02.554846161 +0000 UTC m=+1824.062037378"
Nov 01 12:53:02.718743 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:02.718667    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:05.003701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.002417    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"]
Nov 01 12:53:05.011623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.011594    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.059839 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.059807    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"]
Nov 01 12:53:05.061173 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.061149    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:53:05.061384 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.061360    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:53:05.152903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.152861    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27t65\" (UniqueName: \"kubernetes.io/projected/492450c1-08ca-4640-a328-0abe011d8ff2-kube-api-access-27t65\") pod \"maas-api-revoke-keys-e2e-mt-47feb945-ggbcd\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.153120 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.153099    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-47feb945-ggbcd\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.153395 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.153140    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-47feb945-ggbcd\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.254262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.254165    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27t65\" (UniqueName: \"kubernetes.io/projected/492450c1-08ca-4640-a328-0abe011d8ff2-kube-api-access-27t65\") pod \"maas-api-revoke-keys-e2e-mt-47feb945-ggbcd\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.254434 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.254345    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-47feb945-ggbcd\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.254434 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.254385    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-47feb945-ggbcd\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.255180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.255157    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-47feb945-ggbcd\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.256691 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.256667    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-47feb945-ggbcd\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.308112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.308079    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27t65\" (UniqueName: \"kubernetes.io/projected/492450c1-08ca-4640-a328-0abe011d8ff2-kube-api-access-27t65\") pod \"maas-api-revoke-keys-e2e-mt-47feb945-ggbcd\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.327736 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.327708    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:05.720823 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:05.720795    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"]
Nov 01 12:53:05.726430 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:53:05.726398    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492450c1_08ca_4640_a328_0abe011d8ff2.slice/crio-71e20f1aa43eee5e4790cc537e1e63e7b99ea096c0b48a46a3b805a2f62e626f WatchSource:0}: Error finding container 71e20f1aa43eee5e4790cc537e1e63e7b99ea096c0b48a46a3b805a2f62e626f: Status 404 returned error can't find the container with id 71e20f1aa43eee5e4790cc537e1e63e7b99ea096c0b48a46a3b805a2f62e626f
Nov 01 12:53:06.480870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:06.480832    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd" event={"ID":"492450c1-08ca-4640-a328-0abe011d8ff2","Type":"ContainerStarted","Data":"697b958f1ba7d8abda7d44571353f126994902c73f06452016527518a96ae9d5"}
Nov 01 12:53:06.481378 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:06.480886    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd" event={"ID":"492450c1-08ca-4640-a328-0abe011d8ff2","Type":"ContainerStarted","Data":"71e20f1aa43eee5e4790cc537e1e63e7b99ea096c0b48a46a3b805a2f62e626f"}
Nov 01 12:53:06.588816 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:06.588763    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd" podStartSLOduration=2.588747317 podStartE2EDuration="2.588747317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:06.587405193 +0000 UTC m=+1828.094596412" watchObservedRunningTime="2026-11-01 12:53:06.588747317 +0000 UTC m=+1828.095938534"
Nov 01 12:53:07.449776 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:07.449745    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:53:07.489772 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:07.489736    5725 generic.go:356] "Generic (PLEG): container finished" podID="492450c1-08ca-4640-a328-0abe011d8ff2" containerID="697b958f1ba7d8abda7d44571353f126994902c73f06452016527518a96ae9d5" exitCode=0
Nov 01 12:53:07.490218 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:07.489823    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd" event={"ID":"492450c1-08ca-4640-a328-0abe011d8ff2","Type":"ContainerDied","Data":"697b958f1ba7d8abda7d44571353f126994902c73f06452016527518a96ae9d5"}
Nov 01 12:53:08.673462 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.673432    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:08.799956 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.799863    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-openshift-ca\") pod \"492450c1-08ca-4640-a328-0abe011d8ff2\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") "
Nov 01 12:53:08.800166 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.800125    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492450c1-08ca-4640-a328-0abe011d8ff2-kube-api-access-27t65\" (UniqueName: \"kubernetes.io/projected/492450c1-08ca-4640-a328-0abe011d8ff2-kube-api-access-27t65\") pod \"492450c1-08ca-4640-a328-0abe011d8ff2\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") "
Nov 01 12:53:08.800251 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.800164    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-xks-ca\") pod \"492450c1-08ca-4640-a328-0abe011d8ff2\" (UID: \"492450c1-08ca-4640-a328-0abe011d8ff2\") "
Nov 01 12:53:08.800311 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.800261    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-openshift-ca" pod "492450c1-08ca-4640-a328-0abe011d8ff2" (UID: "492450c1-08ca-4640-a328-0abe011d8ff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:53:08.800704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.800682    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:08.802450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.802316    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-xks-ca" pod "492450c1-08ca-4640-a328-0abe011d8ff2" (UID: "492450c1-08ca-4640-a328-0abe011d8ff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:53:08.802450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.802414    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492450c1-08ca-4640-a328-0abe011d8ff2-kube-api-access-27t65" pod "492450c1-08ca-4640-a328-0abe011d8ff2" (UID: "492450c1-08ca-4640-a328-0abe011d8ff2"). InnerVolumeSpecName "kube-api-access-27t65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:08.901716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.901682    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27t65\" (UniqueName: \"kubernetes.io/projected/492450c1-08ca-4640-a328-0abe011d8ff2-kube-api-access-27t65\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:08.901716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:08.901712    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492450c1-08ca-4640-a328-0abe011d8ff2-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:09.509234 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:09.509199    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"
Nov 01 12:53:09.509511 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:09.509480    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd" event={"ID":"492450c1-08ca-4640-a328-0abe011d8ff2","Type":"ContainerDied","Data":"71e20f1aa43eee5e4790cc537e1e63e7b99ea096c0b48a46a3b805a2f62e626f"}
Nov 01 12:53:09.509603 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:09.509564    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71e20f1aa43eee5e4790cc537e1e63e7b99ea096c0b48a46a3b805a2f62e626f"
Nov 01 12:53:10.000971 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.000936    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"]
Nov 01 12:53:10.001935 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.001912    5725 cpu_manager.go:414] "containerMap: removing container" podUID="492450c1-08ca-4640-a328-0abe011d8ff2" containerName="revoke-keys"
Nov 01 12:53:10.002094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.001945    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492450c1-08ca-4640-a328-0abe011d8ff2" containerName="revoke-keys"
Nov 01 12:53:10.002156 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.002122    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="492450c1-08ca-4640-a328-0abe011d8ff2" containerName="revoke-keys"
Nov 01 12:53:10.005941 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.005919    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.015843 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.015806    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-metrics-certs\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.015984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.015904    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58501f10-37c4-4d76-acda-ae7131ea52fc-openshift-service-ca\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.016078 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.015991    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.016146 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.016080    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9n2x\" (UniqueName: \"kubernetes.io/projected/58501f10-37c4-4d76-acda-ae7131ea52fc-kube-api-access-z9n2x\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.086908 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.086871    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-94c13a-serving-cert\""
Nov 01 12:53:10.108645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.108609    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"]
Nov 01 12:53:10.117444 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.117408    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.117605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.117519    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9n2x\" (UniqueName: \"kubernetes.io/projected/58501f10-37c4-4d76-acda-ae7131ea52fc-kube-api-access-z9n2x\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.117605 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:53:10.117561    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-serving-cert: secret "maas-api-e2e-api-a-94c13a-serving-cert" not found
Nov 01 12:53:10.117815 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:53:10.117629    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls podName:58501f10-37c4-4d76-acda-ae7131ea52fc nodeName:}" failed. No retries permitted until 2026-11-01 12:53:10.617607426 +0000 UTC m=+1832.124798634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls") pod "maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk" (UID: "58501f10-37c4-4d76-acda-ae7131ea52fc") : secret "maas-api-e2e-api-a-94c13a-serving-cert" not found
Nov 01 12:53:10.117815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.117717    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-metrics-certs\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.117815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.117804    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58501f10-37c4-4d76-acda-ae7131ea52fc-openshift-service-ca\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.118635 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.118610    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58501f10-37c4-4d76-acda-ae7131ea52fc-openshift-service-ca\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.141514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.141479    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-metrics-certs\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.173514 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.173480    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9n2x\" (UniqueName: \"kubernetes.io/projected/58501f10-37c4-4d76-acda-ae7131ea52fc-kube-api-access-z9n2x\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.633518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.633484    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.636228 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.636199    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls\") pod \"maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:10.922197 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:10.922116    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:11.313336 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:11.313295    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"]
Nov 01 12:53:11.318484 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:53:11.318450    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58501f10_37c4_4d76_acda_ae7131ea52fc.slice/crio-dc55b2fe0dbc40b43029c9d28cd5a352846f8e1db454882efeb184bfe017e162 WatchSource:0}: Error finding container dc55b2fe0dbc40b43029c9d28cd5a352846f8e1db454882efeb184bfe017e162: Status 404 returned error can't find the container with id dc55b2fe0dbc40b43029c9d28cd5a352846f8e1db454882efeb184bfe017e162
Nov 01 12:53:11.433840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:11.433809    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:53:11.434027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:11.433927    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:53:11.525608 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:11.525571    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk" event={"ID":"58501f10-37c4-4d76-acda-ae7131ea52fc","Type":"ContainerStarted","Data":"dc55b2fe0dbc40b43029c9d28cd5a352846f8e1db454882efeb184bfe017e162"}
Nov 01 12:53:11.717838 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:11.717754    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:11.731041 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:11.731015    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:12.539117 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:12.539072    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk" event={"ID":"58501f10-37c4-4d76-acda-ae7131ea52fc","Type":"ContainerStarted","Data":"5de8e0af67515248588d1fd543ec02de818f445e259bd13ab9a63401074f793a"}
Nov 01 12:53:12.539608 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:12.539155    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:12.551496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:12.551470    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 12:53:12.652257 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:12.652155    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk" podStartSLOduration=2.641545567 podStartE2EDuration="3.652141618s" totalImagesPullingTime="1.010596051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:12.648428829 +0000 UTC m=+1834.155620047" watchObservedRunningTime="2026-11-01 12:53:12.652141618 +0000 UTC m=+1834.159332835"
Nov 01 12:53:14.053658 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:14.053622    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"]
Nov 01 12:53:14.100305 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:14.100268    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9690124c-jl968"]
Nov 01 12:53:14.620105 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:14.620069    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74866cb4-c8da-49fe-a0d0-6f46ea8a87fa" path="/var/lib/kubelet/pods/74866cb4-c8da-49fe-a0d0-6f46ea8a87fa/volumes"
Nov 01 12:53:18.559295 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:18.559263    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 12:53:20.904630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:20.904596    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"]
Nov 01 12:53:20.910397 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:20.910368    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:20.960249 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:20.960216    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c4bbec-serving-cert\""
Nov 01 12:53:21.006530 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.006497    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.006727 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.006585    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzx5\" (UniqueName: \"kubernetes.io/projected/e53102bf-61e4-44c5-a4a1-a15ed898a623-kube-api-access-rtzx5\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.006800 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.006740    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.006915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.006851    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53102bf-61e4-44c5-a4a1-a15ed898a623-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.011090 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.011042    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"]
Nov 01 12:53:21.108629 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.108595    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.108809 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.108656    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzx5\" (UniqueName: \"kubernetes.io/projected/e53102bf-61e4-44c5-a4a1-a15ed898a623-kube-api-access-rtzx5\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.108809 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.108728    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.108809 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:53:21.108757    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-serving-cert: secret "maas-api-e2e-shared-b-w2-c4bbec-serving-cert" not found
Nov 01 12:53:21.108809 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.108776    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53102bf-61e4-44c5-a4a1-a15ed898a623-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.109026 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:53:21.108837    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls podName:e53102bf-61e4-44c5-a4a1-a15ed898a623 nodeName:}" failed. No retries permitted until 2026-11-01 12:53:21.608821872 +0000 UTC m=+1843.116013072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns" (UID: "e53102bf-61e4-44c5-a4a1-a15ed898a623") : secret "maas-api-e2e-shared-b-w2-c4bbec-serving-cert" not found
Nov 01 12:53:21.109452 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.109429    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53102bf-61e4-44c5-a4a1-a15ed898a623-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.111259 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.111238    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.162798 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.162733    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzx5\" (UniqueName: \"kubernetes.io/projected/e53102bf-61e4-44c5-a4a1-a15ed898a623-kube-api-access-rtzx5\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.615608 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.615575    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.618250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.618223    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:21.824587 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:21.824554    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:22.212473 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:22.212397    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"]
Nov 01 12:53:22.217950 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:53:22.217918    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53102bf_61e4_44c5_a4a1_a15ed898a623.slice/crio-e35446dbee67c6e40ce358f0aa557dc78ffd251c956ef3eef3f5e5d752b8ebab WatchSource:0}: Error finding container e35446dbee67c6e40ce358f0aa557dc78ffd251c956ef3eef3f5e5d752b8ebab: Status 404 returned error can't find the container with id e35446dbee67c6e40ce358f0aa557dc78ffd251c956ef3eef3f5e5d752b8ebab
Nov 01 12:53:22.629682 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:22.629632    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns" event={"ID":"e53102bf-61e4-44c5-a4a1-a15ed898a623","Type":"ContainerStarted","Data":"e35446dbee67c6e40ce358f0aa557dc78ffd251c956ef3eef3f5e5d752b8ebab"}
Nov 01 12:53:23.641022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:23.640979    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns" event={"ID":"e53102bf-61e4-44c5-a4a1-a15ed898a623","Type":"ContainerStarted","Data":"b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66"}
Nov 01 12:53:23.641480 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:23.641207    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:23.753995 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:23.753930    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns" podStartSLOduration=2.895395862 podStartE2EDuration="3.75391236s" totalImagesPullingTime="858.516498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:53:23.752355018 +0000 UTC m=+1845.259546239" watchObservedRunningTime="2026-11-01 12:53:23.75391236 +0000 UTC m=+1845.261103577"
Nov 01 12:53:29.656207 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:29.656177    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:53:31.031751 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:53:31.031701    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:53:35.913212 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:35.913170    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"]
Nov 01 12:53:35.913779 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:35.913746    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy" containerID="cri-o://e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848" gracePeriod=30
Nov 01 12:53:41.251270 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.251125    5725 scope.go:122] "RemoveContainer" containerID="d37002ae37ea341de9589dbeb04ce9b4c1f101b03c3a6f677b9e2d5e2d8caac9"
Nov 01 12:53:41.273187 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.272621    5725 scope.go:122] "RemoveContainer" containerID="fc5ff02e2f0f1eb5b28952136a92a35cd6b81e8a99982c70a4affca2034b74c8"
Nov 01 12:53:41.288959 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.288577    5725 scope.go:122] "RemoveContainer" containerID="80e3dba2b01c6d5a8be9cfd3722db5bdc50886f9a7189225b997efad7c54d3ec"
Nov 01 12:53:41.324581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.324557    5725 scope.go:122] "RemoveContainer" containerID="935c6956275c4c3f5aedb01fb0e224cecb634ddb23dbb1b4781bd69f01cd41ce"
Nov 01 12:53:41.326736 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.326716    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:53:41.483365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483319    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b738d395-2a03-407d-8846-459e221674ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b738d395-2a03-407d-8846-459e221674ba-istio-podinfo\") pod \"b738d395-2a03-407d-8846-459e221674ba\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") "
Nov 01 12:53:41.483574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483387    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-istio-token\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-istio-token\") pod \"b738d395-2a03-407d-8846-459e221674ba\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") "
Nov 01 12:53:41.483574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483417    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-envoy\") pod \"b738d395-2a03-407d-8846-459e221674ba\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") "
Nov 01 12:53:41.483574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483439    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-certs\") pod \"b738d395-2a03-407d-8846-459e221674ba\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") "
Nov 01 12:53:41.483574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483464    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-data\") pod \"b738d395-2a03-407d-8846-459e221674ba\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") "
Nov 01 12:53:41.483574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483541    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-socket\") pod \"b738d395-2a03-407d-8846-459e221674ba\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") "
Nov 01 12:53:41.483574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483568    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-credential-socket\") pod \"b738d395-2a03-407d-8846-459e221674ba\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") "
Nov 01 12:53:41.483903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483695    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-kube-api-access-sl9qp\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-kube-api-access-sl9qp\") pod \"b738d395-2a03-407d-8846-459e221674ba\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") "
Nov 01 12:53:41.483903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483758    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b738d395-2a03-407d-8846-459e221674ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b738d395-2a03-407d-8846-459e221674ba-istiod-ca-cert\") pod \"b738d395-2a03-407d-8846-459e221674ba\" (UID: \"b738d395-2a03-407d-8846-459e221674ba\") "
Nov 01 12:53:41.484015 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.483969    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-data" pod "b738d395-2a03-407d-8846-459e221674ba" (UID: "b738d395-2a03-407d-8846-459e221674ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.484334 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.484299    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b738d395-2a03-407d-8846-459e221674ba-istiod-ca-cert" pod "b738d395-2a03-407d-8846-459e221674ba" (UID: "b738d395-2a03-407d-8846-459e221674ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:53:41.484483 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.484454    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-certs" pod "b738d395-2a03-407d-8846-459e221674ba" (UID: "b738d395-2a03-407d-8846-459e221674ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.484547 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.484506    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-credential-socket" pod "b738d395-2a03-407d-8846-459e221674ba" (UID: "b738d395-2a03-407d-8846-459e221674ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.484611 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.484558    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b738d395-2a03-407d-8846-459e221674ba-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.484611 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.484571    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.484717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.484684    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-socket" pod "b738d395-2a03-407d-8846-459e221674ba" (UID: "b738d395-2a03-407d-8846-459e221674ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.486251 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.486209    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b738d395-2a03-407d-8846-459e221674ba-istio-podinfo" pod "b738d395-2a03-407d-8846-459e221674ba" (UID: "b738d395-2a03-407d-8846-459e221674ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:53:41.486822 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.486793    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-istio-token" pod "b738d395-2a03-407d-8846-459e221674ba" (UID: "b738d395-2a03-407d-8846-459e221674ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:41.486946 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.486910    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-kube-api-access-sl9qp" pod "b738d395-2a03-407d-8846-459e221674ba" (UID: "b738d395-2a03-407d-8846-459e221674ba"). InnerVolumeSpecName "kube-api-access-sl9qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:41.487377 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.487359    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-envoy" pod "b738d395-2a03-407d-8846-459e221674ba" (UID: "b738d395-2a03-407d-8846-459e221674ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.589968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.589658    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b738d395-2a03-407d-8846-459e221674ba-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.589968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.589696    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.589968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.589710    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.589968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.589724    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.589968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.589738    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.589968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.589751    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b738d395-2a03-407d-8846-459e221674ba-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.589968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.589764    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl9qp\" (UniqueName: \"kubernetes.io/projected/b738d395-2a03-407d-8846-459e221674ba-kube-api-access-sl9qp\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.845459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.845372    5725 generic.go:356] "Generic (PLEG): container finished" podID="b738d395-2a03-407d-8846-459e221674ba" containerID="e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848" exitCode=0
Nov 01 12:53:41.845459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.845426    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"
Nov 01 12:53:41.845459 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.845451    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" event={"ID":"b738d395-2a03-407d-8846-459e221674ba","Type":"ContainerDied","Data":"e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848"}
Nov 01 12:53:41.845739 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.845511    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" event={"ID":"b738d395-2a03-407d-8846-459e221674ba","Type":"ContainerDied","Data":"ce1d63aff79870f49dd17882a962cdf78aabe786374219921897335fa5b3bcd5"}
Nov 01 12:53:41.845739 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.845546    5725 scope.go:122] "RemoveContainer" containerID="e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848"
Nov 01 12:53:41.859172 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.859151    5725 scope.go:122] "RemoveContainer" containerID="e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848"
Nov 01 12:53:41.859778 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:53:41.859750    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848\": container with ID starting with e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848 not found: ID does not exist" containerID="e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848"
Nov 01 12:53:41.859952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:41.859787    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848"} err="failed to get container status \"e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848\": rpc error: code = NotFound desc = could not find container \"e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848\": container with ID starting with e8c29e6ba9395a02dce713a0f5187b9c7e6d524d81e3aa5959578390e9ca0848 not found: ID does not exist"
Nov 01 12:53:42.010864 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:42.010830    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"]
Nov 01 12:53:42.037562 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:42.028076    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l"]
Nov 01 12:53:42.225380 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:42.225346    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:53:42.225540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:42.225428    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-c9n2l" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:53:42.620531 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:42.620449    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b738d395-2a03-407d-8846-459e221674ba" path="/var/lib/kubelet/pods/b738d395-2a03-407d-8846-459e221674ba/volumes"
Nov 01 12:53:43.620821 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:43.620788    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": read tcp 10.132.0.2:54098->10.132.0.171:15021: read: connection reset by peer" start-of-body=
Nov 01 12:53:43.621241 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:43.620859    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": read tcp 10.132.0.2:54098->10.132.0.171:15021: read: connection reset by peer"
Nov 01 12:53:43.869264 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:43.869164    5725 generic.go:356] "Generic (PLEG): container finished" podID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerID="e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a" exitCode=0
Nov 01 12:53:43.869571 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:43.869304    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" event={"ID":"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9","Type":"ContainerDied","Data":"e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a"}
Nov 01 12:53:43.869877 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:43.869857    5725 scope.go:122] "RemoveContainer" containerID="e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a"
Nov 01 12:53:43.871935 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:43.871896    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:53:43.872039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:43.872008    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:53:43.872149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:43.872075    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 12:53:44.880031 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:44.879988    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" event={"ID":"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9","Type":"ContainerStarted","Data":"20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26"}
Nov 01 12:53:45.183411 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:45.183322    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:53:45.183574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:45.183490    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 12:53:45.183574 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:45.183539    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 12:53:46.184382 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:46.184341    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 12:53:46.184824 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:46.184406    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 12:53:47.183681 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:47.183641    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 12:53:47.183879 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:47.183697    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 12:53:48.184097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:48.184064    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 12:53:48.184495 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:48.184127    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 12:53:49.183651 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:49.183615    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 12:53:49.183878 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:49.183669    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 12:53:50.184254 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:50.184214    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 01 12:53:50.184660 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:50.184284    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 01 12:53:51.187645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:51.187612    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:53:51.188175 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:51.188149    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:53:51.188828 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:53:51.188806    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:54:01.030254 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:54:01.030212    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:54:05.727665 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:05.727601    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"]
Nov 01 12:54:05.728255 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:05.727865    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy" containerID="cri-o://d11645f3d35f1ef8a2f2fa79969d3d63646ca91f1edccab0bf2b94fe1309f33e" gracePeriod=30
Nov 01 12:54:05.775901 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:05.775853    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"]
Nov 01 12:54:05.776385 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:05.776349    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy" containerID="cri-o://01cc0820b4fdfcfd507fcfcdbc9c78313d306321b9b02aff98d9d9ea3f47b026" gracePeriod=30
Nov 01 12:54:05.780963 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:05.780933    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"]
Nov 01 12:54:05.781161 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:05.781138    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy" containerID="cri-o://1e56258900e2a58adcc35be42accb200f24c050de049cc5d1c8a439394d86ab4" gracePeriod=30
Nov 01 12:54:11.193125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.190740    5725 generic.go:356] "Generic (PLEG): container finished" podID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerID="01cc0820b4fdfcfd507fcfcdbc9c78313d306321b9b02aff98d9d9ea3f47b026" exitCode=0
Nov 01 12:54:11.193125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.190955    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" event={"ID":"dd50dc72-ea77-4b81-9be5-db8ff0c0978e","Type":"ContainerDied","Data":"01cc0820b4fdfcfd507fcfcdbc9c78313d306321b9b02aff98d9d9ea3f47b026"}
Nov 01 12:54:11.194955 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.194927    5725 generic.go:356] "Generic (PLEG): container finished" podID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerID="d11645f3d35f1ef8a2f2fa79969d3d63646ca91f1edccab0bf2b94fe1309f33e" exitCode=0
Nov 01 12:54:11.195271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.195246    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" event={"ID":"71d86828-3b34-49c4-82d9-8d90a1908e65","Type":"ContainerDied","Data":"d11645f3d35f1ef8a2f2fa79969d3d63646ca91f1edccab0bf2b94fe1309f33e"}
Nov 01 12:54:11.199135 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.197348    5725 generic.go:356] "Generic (PLEG): container finished" podID="0bb3faca-0aac-4627-9b21-759db7314974" containerID="1e56258900e2a58adcc35be42accb200f24c050de049cc5d1c8a439394d86ab4" exitCode=0
Nov 01 12:54:11.199135 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.197421    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" event={"ID":"0bb3faca-0aac-4627-9b21-759db7314974","Type":"ContainerDied","Data":"1e56258900e2a58adcc35be42accb200f24c050de049cc5d1c8a439394d86ab4"}
Nov 01 12:54:11.288019 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.287982    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:54:11.368080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.367260    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:54:11.380601 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.380562    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-credential-socket\") pod \"71d86828-3b34-49c4-82d9-8d90a1908e65\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") "
Nov 01 12:54:11.380806 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.380637    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-token\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-token\") pod \"71d86828-3b34-49c4-82d9-8d90a1908e65\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") "
Nov 01 12:54:11.380806 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.380696    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-envoy\") pod \"71d86828-3b34-49c4-82d9-8d90a1908e65\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") "
Nov 01 12:54:11.380806 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.380731    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-socket\") pod \"71d86828-3b34-49c4-82d9-8d90a1908e65\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") "
Nov 01 12:54:11.380806 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.380771    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-podinfo\") pod \"71d86828-3b34-49c4-82d9-8d90a1908e65\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") "
Nov 01 12:54:11.381021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.380808    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-certs\") pod \"71d86828-3b34-49c4-82d9-8d90a1908e65\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") "
Nov 01 12:54:11.381021 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.381010    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-kube-api-access-5lrxl\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-kube-api-access-5lrxl\") pod \"71d86828-3b34-49c4-82d9-8d90a1908e65\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") "
Nov 01 12:54:11.381168 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.381105    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-data\") pod \"71d86828-3b34-49c4-82d9-8d90a1908e65\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") "
Nov 01 12:54:11.381168 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.381158    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d86828-3b34-49c4-82d9-8d90a1908e65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d86828-3b34-49c4-82d9-8d90a1908e65-istiod-ca-cert\") pod \"71d86828-3b34-49c4-82d9-8d90a1908e65\" (UID: \"71d86828-3b34-49c4-82d9-8d90a1908e65\") "
Nov 01 12:54:11.382104 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.382074    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d86828-3b34-49c4-82d9-8d90a1908e65-istiod-ca-cert" pod "71d86828-3b34-49c4-82d9-8d90a1908e65" (UID: "71d86828-3b34-49c4-82d9-8d90a1908e65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:54:11.382367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.382345    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-credential-socket" pod "71d86828-3b34-49c4-82d9-8d90a1908e65" (UID: "71d86828-3b34-49c4-82d9-8d90a1908e65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.383268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.382992    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-socket" pod "71d86828-3b34-49c4-82d9-8d90a1908e65" (UID: "71d86828-3b34-49c4-82d9-8d90a1908e65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.383268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.383232    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-data" pod "71d86828-3b34-49c4-82d9-8d90a1908e65" (UID: "71d86828-3b34-49c4-82d9-8d90a1908e65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.383535 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.383500    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-certs" pod "71d86828-3b34-49c4-82d9-8d90a1908e65" (UID: "71d86828-3b34-49c4-82d9-8d90a1908e65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.385623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.385587    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-podinfo" pod "71d86828-3b34-49c4-82d9-8d90a1908e65" (UID: "71d86828-3b34-49c4-82d9-8d90a1908e65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:54:11.388121 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.385936    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-envoy" pod "71d86828-3b34-49c4-82d9-8d90a1908e65" (UID: "71d86828-3b34-49c4-82d9-8d90a1908e65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.388121 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.386778    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-token" pod "71d86828-3b34-49c4-82d9-8d90a1908e65" (UID: "71d86828-3b34-49c4-82d9-8d90a1908e65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:11.388121 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.387370    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-kube-api-access-5lrxl" pod "71d86828-3b34-49c4-82d9-8d90a1908e65" (UID: "71d86828-3b34-49c4-82d9-8d90a1908e65"). InnerVolumeSpecName "kube-api-access-5lrxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:11.401144 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.401121    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:54:11.483038 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.482997    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istiod-ca-cert\") pod \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") "
Nov 01 12:54:11.483293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483099    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bb3faca-0aac-4627-9b21-759db7314974-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb3faca-0aac-4627-9b21-759db7314974-istio-podinfo\") pod \"0bb3faca-0aac-4627-9b21-759db7314974\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") "
Nov 01 12:54:11.483293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483197    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-certs\") pod \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") "
Nov 01 12:54:11.483293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483235    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-envoy\") pod \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") "
Nov 01 12:54:11.483293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483271    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-podinfo\") pod \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") "
Nov 01 12:54:11.483521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483312    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-data\") pod \"0bb3faca-0aac-4627-9b21-759db7314974\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") "
Nov 01 12:54:11.483521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483370    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-credential-socket\") pod \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") "
Nov 01 12:54:11.483521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483402    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istiod-ca-cert" pod "dd50dc72-ea77-4b81-9be5-db8ff0c0978e" (UID: "dd50dc72-ea77-4b81-9be5-db8ff0c0978e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:54:11.483521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483414    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-kube-api-access-vtwvs\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-kube-api-access-vtwvs\") pod \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") "
Nov 01 12:54:11.483521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483450    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-credential-socket\") pod \"0bb3faca-0aac-4627-9b21-759db7314974\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") "
Nov 01 12:54:11.483785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483545    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-token\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-token\") pod \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") "
Nov 01 12:54:11.483785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483576    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb3faca-0aac-4627-9b21-759db7314974-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3faca-0aac-4627-9b21-759db7314974-istiod-ca-cert\") pod \"0bb3faca-0aac-4627-9b21-759db7314974\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") "
Nov 01 12:54:11.483785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483634    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-certs\") pod \"0bb3faca-0aac-4627-9b21-759db7314974\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") "
Nov 01 12:54:11.483785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483659    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-certs" pod "dd50dc72-ea77-4b81-9be5-db8ff0c0978e" (UID: "dd50dc72-ea77-4b81-9be5-db8ff0c0978e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.483785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483712    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-data\") pod \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") "
Nov 01 12:54:11.483785 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483759    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-socket\") pod \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\" (UID: \"dd50dc72-ea77-4b81-9be5-db8ff0c0978e\") "
Nov 01 12:54:11.484128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483808    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-kube-api-access-mtz5f\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-kube-api-access-mtz5f\") pod \"0bb3faca-0aac-4627-9b21-759db7314974\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") "
Nov 01 12:54:11.484128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483842    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-socket\") pod \"0bb3faca-0aac-4627-9b21-759db7314974\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") "
Nov 01 12:54:11.484128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483900    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-istio-token\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-istio-token\") pod \"0bb3faca-0aac-4627-9b21-759db7314974\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") "
Nov 01 12:54:11.484128 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.483959    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-envoy\") pod \"0bb3faca-0aac-4627-9b21-759db7314974\" (UID: \"0bb3faca-0aac-4627-9b21-759db7314974\") "
Nov 01 12:54:11.484536 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484517    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484538    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484554    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484568    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484582    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484596    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrxl\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-kube-api-access-5lrxl\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484609    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484623    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d86828-3b34-49c4-82d9-8d90a1908e65-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484636    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484651    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d86828-3b34-49c4-82d9-8d90a1908e65-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.484986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.484664    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d86828-3b34-49c4-82d9-8d90a1908e65-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.486097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.485411    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-socket" pod "dd50dc72-ea77-4b81-9be5-db8ff0c0978e" (UID: "dd50dc72-ea77-4b81-9be5-db8ff0c0978e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.486097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.485487    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-certs" pod "0bb3faca-0aac-4627-9b21-759db7314974" (UID: "0bb3faca-0aac-4627-9b21-759db7314974"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.486097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.485605    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-data" pod "dd50dc72-ea77-4b81-9be5-db8ff0c0978e" (UID: "dd50dc72-ea77-4b81-9be5-db8ff0c0978e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.486097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.485877    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-data" pod "0bb3faca-0aac-4627-9b21-759db7314974" (UID: "0bb3faca-0aac-4627-9b21-759db7314974"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.486097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.485994    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-credential-socket" pod "dd50dc72-ea77-4b81-9be5-db8ff0c0978e" (UID: "dd50dc72-ea77-4b81-9be5-db8ff0c0978e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.486097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.486084    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb3faca-0aac-4627-9b21-759db7314974-istiod-ca-cert" pod "0bb3faca-0aac-4627-9b21-759db7314974" (UID: "0bb3faca-0aac-4627-9b21-759db7314974"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:54:11.486492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.486099    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-credential-socket" pod "0bb3faca-0aac-4627-9b21-759db7314974" (UID: "0bb3faca-0aac-4627-9b21-759db7314974"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.486492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.486258    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-socket" pod "0bb3faca-0aac-4627-9b21-759db7314974" (UID: "0bb3faca-0aac-4627-9b21-759db7314974"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.486892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.486834    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-envoy" pod "dd50dc72-ea77-4b81-9be5-db8ff0c0978e" (UID: "dd50dc72-ea77-4b81-9be5-db8ff0c0978e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.486892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.486858    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bb3faca-0aac-4627-9b21-759db7314974-istio-podinfo" pod "0bb3faca-0aac-4627-9b21-759db7314974" (UID: "0bb3faca-0aac-4627-9b21-759db7314974"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:54:11.487151 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.487129    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-podinfo" pod "dd50dc72-ea77-4b81-9be5-db8ff0c0978e" (UID: "dd50dc72-ea77-4b81-9be5-db8ff0c0978e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:54:11.487278 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.487262    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-envoy" pod "0bb3faca-0aac-4627-9b21-759db7314974" (UID: "0bb3faca-0aac-4627-9b21-759db7314974"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:11.488617 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.488580    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-token" pod "dd50dc72-ea77-4b81-9be5-db8ff0c0978e" (UID: "dd50dc72-ea77-4b81-9be5-db8ff0c0978e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:11.488719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.488612    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-kube-api-access-mtz5f" pod "0bb3faca-0aac-4627-9b21-759db7314974" (UID: "0bb3faca-0aac-4627-9b21-759db7314974"). InnerVolumeSpecName "kube-api-access-mtz5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:11.488937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.488915    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-kube-api-access-vtwvs" pod "dd50dc72-ea77-4b81-9be5-db8ff0c0978e" (UID: "dd50dc72-ea77-4b81-9be5-db8ff0c0978e"). InnerVolumeSpecName "kube-api-access-vtwvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:11.489170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.489150    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-istio-token" pod "0bb3faca-0aac-4627-9b21-759db7314974" (UID: "0bb3faca-0aac-4627-9b21-759db7314974"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:11.587357 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587320    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtwvs\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-kube-api-access-vtwvs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587357 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587356    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587371    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587383    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb3faca-0aac-4627-9b21-759db7314974-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587395    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587407    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587418    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587431    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtz5f\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-kube-api-access-mtz5f\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587444    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587456    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb3faca-0aac-4627-9b21-759db7314974-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587468    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587482    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb3faca-0aac-4627-9b21-759db7314974-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587495    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587509    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587520    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb3faca-0aac-4627-9b21-759db7314974-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:11.587602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:11.587532    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd50dc72-ea77-4b81-9be5-db8ff0c0978e-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:12.207787 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.207705    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"
Nov 01 12:54:12.207787 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.207755    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568" event={"ID":"71d86828-3b34-49c4-82d9-8d90a1908e65","Type":"ContainerDied","Data":"0f44cf0986caa1cf153c080dee96418b8dac5fa92dfc520b0e96d6601eada3fd"}
Nov 01 12:54:12.208327 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.207820    5725 scope.go:122] "RemoveContainer" containerID="d11645f3d35f1ef8a2f2fa79969d3d63646ca91f1edccab0bf2b94fe1309f33e"
Nov 01 12:54:12.209640 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.209617    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"
Nov 01 12:54:12.209761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.209638    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt" event={"ID":"0bb3faca-0aac-4627-9b21-759db7314974","Type":"ContainerDied","Data":"e7049ead99fce29052c97ec5a5c48d840d466863c7352ab31efc05768599a9be"}
Nov 01 12:54:12.211483 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.211462    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"
Nov 01 12:54:12.211623 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.211512    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w" event={"ID":"dd50dc72-ea77-4b81-9be5-db8ff0c0978e","Type":"ContainerDied","Data":"2e9332abc113bdc516d2382ff6979e8955b25a418ee9c0f34dcc95729a3ebc90"}
Nov 01 12:54:12.223941 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.223924    5725 scope.go:122] "RemoveContainer" containerID="1e56258900e2a58adcc35be42accb200f24c050de049cc5d1c8a439394d86ab4"
Nov 01 12:54:12.237944 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.237917    5725 scope.go:122] "RemoveContainer" containerID="01cc0820b4fdfcfd507fcfcdbc9c78313d306321b9b02aff98d9d9ea3f47b026"
Nov 01 12:54:12.365731 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.365619    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"]
Nov 01 12:54:12.412663 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.412629    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b8568"]
Nov 01 12:54:12.474405 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.474367    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"]
Nov 01 12:54:12.478801 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.478773    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-sscmt"]
Nov 01 12:54:12.582297 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.582260    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"]
Nov 01 12:54:12.586012 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.585991    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-nbf9w"]
Nov 01 12:54:12.617538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.617508    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb3faca-0aac-4627-9b21-759db7314974" path="/var/lib/kubelet/pods/0bb3faca-0aac-4627-9b21-759db7314974/volumes"
Nov 01 12:54:12.618039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.618024    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" path="/var/lib/kubelet/pods/71d86828-3b34-49c4-82d9-8d90a1908e65/volumes"
Nov 01 12:54:12.618588 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:12.618573    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" path="/var/lib/kubelet/pods/dd50dc72-ea77-4b81-9be5-db8ff0c0978e/volumes"
Nov 01 12:54:13.007645 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:13.007600    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"]
Nov 01 12:54:13.013306 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:13.013276    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47feb945-ggbcd"]
Nov 01 12:54:14.617268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:14.617231    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492450c1-08ca-4640-a328-0abe011d8ff2" path="/var/lib/kubelet/pods/492450c1-08ca-4640-a328-0abe011d8ff2/volumes"
Nov 01 12:54:18.310455 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.310418    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"]
Nov 01 12:54:18.311253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311229    5725 cpu_manager.go:414] "containerMap: removing container" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy"
Nov 01 12:54:18.311253 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311250    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311264    5725 cpu_manager.go:414] "containerMap: removing container" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311269    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311281    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311286    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311300    5725 cpu_manager.go:414] "containerMap: removing container" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311305    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311403    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd50dc72-ea77-4b81-9be5-db8ff0c0978e" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311412    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb3faca-0aac-4627-9b21-759db7314974" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311420    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b738d395-2a03-407d-8846-459e221674ba" containerName="istio-proxy"
Nov 01 12:54:18.311435 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.311428    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d86828-3b34-49c4-82d9-8d90a1908e65" containerName="istio-proxy"
Nov 01 12:54:18.316755 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.316736    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.367273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.367236    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"]
Nov 01 12:54:18.391010 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.390972    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da507d04-6a1b-409f-a008-b38baf7bcd06-openshift-service-ca\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.391217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.391021    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqzq\" (UniqueName: \"kubernetes.io/projected/da507d04-6a1b-409f-a008-b38baf7bcd06-kube-api-access-dfqzq\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.391217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.391161    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.391356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.391273    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-metrics-certs\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.409688 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.409665    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f893c2-serving-cert\""
Nov 01 12:54:18.492946 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.492911    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da507d04-6a1b-409f-a008-b38baf7bcd06-openshift-service-ca\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.492946 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.492948    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqzq\" (UniqueName: \"kubernetes.io/projected/da507d04-6a1b-409f-a008-b38baf7bcd06-kube-api-access-dfqzq\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.493243 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.492977    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.493243 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.493039    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-metrics-certs\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.493243 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:54:18.493147    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-serving-cert: secret "maas-api-e2e-api-a-f893c2-serving-cert" not found
Nov 01 12:54:18.493243 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:54:18.493228    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls podName:da507d04-6a1b-409f-a008-b38baf7bcd06 nodeName:}" failed. No retries permitted until 2026-11-01 12:54:18.993208118 +0000 UTC m=+1900.500399329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls") pod "maas-api-e2e-api-a-f893c2-6844566554-52qrh" (UID: "da507d04-6a1b-409f-a008-b38baf7bcd06") : secret "maas-api-e2e-api-a-f893c2-serving-cert" not found
Nov 01 12:54:18.493705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.493682    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da507d04-6a1b-409f-a008-b38baf7bcd06-openshift-service-ca\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.495510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.495491    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-metrics-certs\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:18.547871 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:18.547836    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqzq\" (UniqueName: \"kubernetes.io/projected/da507d04-6a1b-409f-a008-b38baf7bcd06-kube-api-access-dfqzq\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:19.000490 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:19.000409    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:19.002809 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:19.002789    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls\") pod \"maas-api-e2e-api-a-f893c2-6844566554-52qrh\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:19.232329 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:19.232293    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:19.613871 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:19.613786    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"]
Nov 01 12:54:19.620278 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:54:19.620246    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda507d04_6a1b_409f_a008_b38baf7bcd06.slice/crio-ae0d50d1c567e5e8bcf84978b818fd36e65345c05cfd86d8b9b25383bf9ef2e7 WatchSource:0}: Error finding container ae0d50d1c567e5e8bcf84978b818fd36e65345c05cfd86d8b9b25383bf9ef2e7: Status 404 returned error can't find the container with id ae0d50d1c567e5e8bcf84978b818fd36e65345c05cfd86d8b9b25383bf9ef2e7
Nov 01 12:54:20.391074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:20.379571    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh" event={"ID":"da507d04-6a1b-409f-a008-b38baf7bcd06","Type":"ContainerStarted","Data":"ae0d50d1c567e5e8bcf84978b818fd36e65345c05cfd86d8b9b25383bf9ef2e7"}
Nov 01 12:54:21.391477 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:21.391439    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh" event={"ID":"da507d04-6a1b-409f-a008-b38baf7bcd06","Type":"ContainerStarted","Data":"693e38d2a6bc3b3a1f0aa8f3b77e3c9f28e590da73bf88a65f1f6391b4467cb8"}
Nov 01 12:54:21.391965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:21.391511    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:21.502286 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:21.502226    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh" podStartSLOduration=2.442924909 podStartE2EDuration="3.502213937s" totalImagesPullingTime="1.059289028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:54:21.499816466 +0000 UTC m=+1903.007007685" watchObservedRunningTime="2026-11-01 12:54:21.502213937 +0000 UTC m=+1903.009405215"
Nov 01 12:54:27.409273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:27.409240    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:54:31.029512 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:54:31.029474    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:54:41.414041 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:54:41.414004    5725 scope.go:122] "RemoveContainer" containerID="67a40b77a952337fd00e9d45a00aadb59f7e4d443d75abdfc8e5b82c3fdc5815"
Nov 01 12:55:01.039404 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:55:01.039362    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:55:22.562675 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.562637    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"]
Nov 01 12:55:22.567370 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.567344    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.616332 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.616304    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6b9030-serving-cert\""
Nov 01 12:55:22.638062 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.638014    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-metrics-certs\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.638239 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.638079    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.638312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.638284    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d13f06-e631-463e-95ba-95ae13f3fa5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.638470 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.638434    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4z28\" (UniqueName: \"kubernetes.io/projected/27d13f06-e631-463e-95ba-95ae13f3fa5e-kube-api-access-r4z28\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.669975 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.669939    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"]
Nov 01 12:55:22.740022 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.739988    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d13f06-e631-463e-95ba-95ae13f3fa5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.741206 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.741181    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4z28\" (UniqueName: \"kubernetes.io/projected/27d13f06-e631-463e-95ba-95ae13f3fa5e-kube-api-access-r4z28\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.741534 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.741516    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-metrics-certs\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.741803 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.741678    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.742059 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.741005    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d13f06-e631-463e-95ba-95ae13f3fa5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.742156 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:55:22.741776    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-serving-cert: secret "maas-api-e2e-api-a-6b9030-serving-cert" not found
Nov 01 12:55:22.742618 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:55:22.742412    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls podName:27d13f06-e631-463e-95ba-95ae13f3fa5e nodeName:}" failed. No retries permitted until 2026-11-01 12:55:23.242390863 +0000 UTC m=+1964.749582062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls") pod "maas-api-e2e-api-a-6b9030-8977f6855-vlbvw" (UID: "27d13f06-e631-463e-95ba-95ae13f3fa5e") : secret "maas-api-e2e-api-a-6b9030-serving-cert" not found
Nov 01 12:55:22.756540 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.756511    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-metrics-certs\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:22.794972 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:22.794941    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4z28\" (UniqueName: \"kubernetes.io/projected/27d13f06-e631-463e-95ba-95ae13f3fa5e-kube-api-access-r4z28\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:23.251609 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:23.251573    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:23.251810 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:55:23.251748    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-serving-cert: secret "maas-api-e2e-api-a-6b9030-serving-cert" not found
Nov 01 12:55:23.251865 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:55:23.251813    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls podName:27d13f06-e631-463e-95ba-95ae13f3fa5e nodeName:}" failed. No retries permitted until 2026-11-01 12:55:24.251796202 +0000 UTC m=+1965.758987403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls") pod "maas-api-e2e-api-a-6b9030-8977f6855-vlbvw" (UID: "27d13f06-e631-463e-95ba-95ae13f3fa5e") : secret "maas-api-e2e-api-a-6b9030-serving-cert" not found
Nov 01 12:55:24.266639 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:24.266601    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:24.269206 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:24.269178    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls\") pod \"maas-api-e2e-api-a-6b9030-8977f6855-vlbvw\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:24.383984 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:24.383950    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:24.763496 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:24.763451    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"]
Nov 01 12:55:24.770356 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:55:24.770325    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d13f06_e631_463e_95ba_95ae13f3fa5e.slice/crio-42fecdded9ab8711a333516b7ae9e28b452d3adca21b259bc49b15b7aba3add3 WatchSource:0}: Error finding container 42fecdded9ab8711a333516b7ae9e28b452d3adca21b259bc49b15b7aba3add3: Status 404 returned error can't find the container with id 42fecdded9ab8711a333516b7ae9e28b452d3adca21b259bc49b15b7aba3add3
Nov 01 12:55:25.063539 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:25.063436    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw" event={"ID":"27d13f06-e631-463e-95ba-95ae13f3fa5e","Type":"ContainerStarted","Data":"42fecdded9ab8711a333516b7ae9e28b452d3adca21b259bc49b15b7aba3add3"}
Nov 01 12:55:26.073736 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:26.073646    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw" event={"ID":"27d13f06-e631-463e-95ba-95ae13f3fa5e","Type":"ContainerStarted","Data":"f1956216e61a3197bef494b4dc8189f583dc27fa6d0ce66d09dd79ce06b5a3f8"}
Nov 01 12:55:26.074169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:26.073736    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:26.194702 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:26.194651    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw" podStartSLOduration=3.213414648 podStartE2EDuration="4.194637921s" totalImagesPullingTime="981.223273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:55:26.192634338 +0000 UTC m=+1967.699825557" watchObservedRunningTime="2026-11-01 12:55:26.194637921 +0000 UTC m=+1967.701829137"
Nov 01 12:55:31.029357 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:55:31.029312    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:55:32.097172 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:32.097141    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:55:38.353358 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:38.353323    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7091-162683"]
Nov 01 12:55:38.357646 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:38.357620    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-162683"
Nov 01 12:55:38.408526 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:38.408496    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-162683"]
Nov 01 12:55:38.444395 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:38.444354    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjt4b\" (UniqueName: \"kubernetes.io/projected/1bb5d696-4939-471e-8ab9-1cdb2d9cd528-kube-api-access-vjt4b\") pod \"test-curl-7091-162683\" (UID: \"1bb5d696-4939-471e-8ab9-1cdb2d9cd528\") " pod="opendatahub/test-curl-7091-162683"
Nov 01 12:55:38.449636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:38.449612    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rbvvl\""
Nov 01 12:55:38.545696 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:38.545663    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjt4b\" (UniqueName: \"kubernetes.io/projected/1bb5d696-4939-471e-8ab9-1cdb2d9cd528-kube-api-access-vjt4b\") pod \"test-curl-7091-162683\" (UID: \"1bb5d696-4939-471e-8ab9-1cdb2d9cd528\") " pod="opendatahub/test-curl-7091-162683"
Nov 01 12:55:38.598976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:38.598947    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjt4b\" (UniqueName: \"kubernetes.io/projected/1bb5d696-4939-471e-8ab9-1cdb2d9cd528-kube-api-access-vjt4b\") pod \"test-curl-7091-162683\" (UID: \"1bb5d696-4939-471e-8ab9-1cdb2d9cd528\") " pod="opendatahub/test-curl-7091-162683"
Nov 01 12:55:38.718753 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:38.718677    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rbvvl\""
Nov 01 12:55:38.720447 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:38.720423    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-162683"
Nov 01 12:55:39.104354 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:39.104322    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-162683"]
Nov 01 12:55:39.110279 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:55:39.110254    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bb5d696_4939_471e_8ab9_1cdb2d9cd528.slice/crio-405e01f9aa02c9e7e38633df8a0c0fc6a93f2420357ad0650be08d318d029e60 WatchSource:0}: Error finding container 405e01f9aa02c9e7e38633df8a0c0fc6a93f2420357ad0650be08d318d029e60: Status 404 returned error can't find the container with id 405e01f9aa02c9e7e38633df8a0c0fc6a93f2420357ad0650be08d318d029e60
Nov 01 12:55:39.186007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:39.185981    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-162683" event={"ID":"1bb5d696-4939-471e-8ab9-1cdb2d9cd528","Type":"ContainerStarted","Data":"405e01f9aa02c9e7e38633df8a0c0fc6a93f2420357ad0650be08d318d029e60"}
Nov 01 12:55:40.194522 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:40.194482    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-162683" event={"ID":"1bb5d696-4939-471e-8ab9-1cdb2d9cd528","Type":"ContainerStarted","Data":"fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc"}
Nov 01 12:55:40.302254 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:40.302194    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7091-162683" podStartSLOduration=2.302179758 podStartE2EDuration="2.302179758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:55:40.300720126 +0000 UTC m=+1981.807911345" watchObservedRunningTime="2026-11-01 12:55:40.302179758 +0000 UTC m=+1981.809370976"
Nov 01 12:55:41.658441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:41.658407    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7091-162683"]
Nov 01 12:55:41.668093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:41.664558    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7091-162683"]
Nov 01 12:55:42.210613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:42.210568    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7091-162683" podUID="1bb5d696-4939-471e-8ab9-1cdb2d9cd528" containerName="test-curl-7091-162683" containerID="cri-o://fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc" gracePeriod=2
Nov 01 12:55:42.258977 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:55:42.258929    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-162683\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="1bb5d696-4939-471e-8ab9-1cdb2d9cd528" pod="opendatahub/test-curl-7091-162683"
Nov 01 12:55:44.502399 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:44.502373    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-162683"
Nov 01 12:55:44.627775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:44.627727    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb5d696-4939-471e-8ab9-1cdb2d9cd528-kube-api-access-vjt4b\" (UniqueName: \"kubernetes.io/projected/1bb5d696-4939-471e-8ab9-1cdb2d9cd528-kube-api-access-vjt4b\") pod \"1bb5d696-4939-471e-8ab9-1cdb2d9cd528\" (UID: \"1bb5d696-4939-471e-8ab9-1cdb2d9cd528\") "
Nov 01 12:55:44.631036 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:44.631006    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb5d696-4939-471e-8ab9-1cdb2d9cd528-kube-api-access-vjt4b" pod "1bb5d696-4939-471e-8ab9-1cdb2d9cd528" (UID: "1bb5d696-4939-471e-8ab9-1cdb2d9cd528"). InnerVolumeSpecName "kube-api-access-vjt4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:55:44.729977 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:44.729901    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjt4b\" (UniqueName: \"kubernetes.io/projected/1bb5d696-4939-471e-8ab9-1cdb2d9cd528-kube-api-access-vjt4b\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:55:45.236596 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:45.236558    5725 generic.go:356] "Generic (PLEG): container finished" podID="1bb5d696-4939-471e-8ab9-1cdb2d9cd528" containerID="fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc" exitCode=137
Nov 01 12:55:45.236797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:45.236603    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-162683"
Nov 01 12:55:45.236797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:45.236664    5725 scope.go:122] "RemoveContainer" containerID="fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc"
Nov 01 12:55:45.248847 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:45.248824    5725 scope.go:122] "RemoveContainer" containerID="fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc"
Nov 01 12:55:45.249133 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:55:45.249111    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc\": container with ID starting with fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc not found: ID does not exist" containerID="fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc"
Nov 01 12:55:45.249233 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:45.249143    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc"} err="failed to get container status \"fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc\": rpc error: code = NotFound desc = could not find container \"fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc\": container with ID starting with fca996ccae5338aa7e5dc245006892b4ec8125d7b0e02ee13e80a9570b1efabc not found: ID does not exist"
Nov 01 12:55:46.618030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:55:46.617997    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb5d696-4939-471e-8ab9-1cdb2d9cd528" path="/var/lib/kubelet/pods/1bb5d696-4939-471e-8ab9-1cdb2d9cd528/volumes"
Nov 01 12:56:01.027812 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:56:01.027716    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:56:05.853710 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:05.853663    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"]
Nov 01 12:56:05.974252 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:05.974205    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"]
Nov 01 12:56:10.854145 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:10.854041    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc" podUID="5b3b02e0-06f8-4865-8301-fc5f33a74972" containerName="payload-processing" containerID="cri-o://1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d" gracePeriod=25
Nov 01 12:56:10.974695 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:10.974652    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg" podUID="33626a56-ce7d-49b8-9f98-316044f8fa80" containerName="payload-pre-processing" containerID="cri-o://d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a" gracePeriod=25
Nov 01 12:56:11.207770 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.207117    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:56:11.363773 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.363736    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:56:11.378716 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.378686    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3b02e0-06f8-4865-8301-fc5f33a74972-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3b02e0-06f8-4865-8301-fc5f33a74972-plugins-config-volume\") pod \"5b3b02e0-06f8-4865-8301-fc5f33a74972\" (UID: \"5b3b02e0-06f8-4865-8301-fc5f33a74972\") "
Nov 01 12:56:11.378891 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.378809    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3b02e0-06f8-4865-8301-fc5f33a74972-kube-api-access-kc7wb\" (UniqueName: \"kubernetes.io/projected/5b3b02e0-06f8-4865-8301-fc5f33a74972-kube-api-access-kc7wb\") pod \"5b3b02e0-06f8-4865-8301-fc5f33a74972\" (UID: \"5b3b02e0-06f8-4865-8301-fc5f33a74972\") "
Nov 01 12:56:11.379341 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.379312    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3b02e0-06f8-4865-8301-fc5f33a74972-plugins-config-volume" pod "5b3b02e0-06f8-4865-8301-fc5f33a74972" (UID: "5b3b02e0-06f8-4865-8301-fc5f33a74972"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:11.379602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.379575    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3b02e0-06f8-4865-8301-fc5f33a74972-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:11.381961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.381934    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3b02e0-06f8-4865-8301-fc5f33a74972-kube-api-access-kc7wb" pod "5b3b02e0-06f8-4865-8301-fc5f33a74972" (UID: "5b3b02e0-06f8-4865-8301-fc5f33a74972"). InnerVolumeSpecName "kube-api-access-kc7wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:11.480737 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.480702    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33626a56-ce7d-49b8-9f98-316044f8fa80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33626a56-ce7d-49b8-9f98-316044f8fa80-plugins-config-volume\") pod \"33626a56-ce7d-49b8-9f98-316044f8fa80\" (UID: \"33626a56-ce7d-49b8-9f98-316044f8fa80\") "
Nov 01 12:56:11.480929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.480818    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33626a56-ce7d-49b8-9f98-316044f8fa80-kube-api-access-j9vml\" (UniqueName: \"kubernetes.io/projected/33626a56-ce7d-49b8-9f98-316044f8fa80-kube-api-access-j9vml\") pod \"33626a56-ce7d-49b8-9f98-316044f8fa80\" (UID: \"33626a56-ce7d-49b8-9f98-316044f8fa80\") "
Nov 01 12:56:11.481322 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.481302    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc7wb\" (UniqueName: \"kubernetes.io/projected/5b3b02e0-06f8-4865-8301-fc5f33a74972-kube-api-access-kc7wb\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:11.483558 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.483527    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33626a56-ce7d-49b8-9f98-316044f8fa80-kube-api-access-j9vml" pod "33626a56-ce7d-49b8-9f98-316044f8fa80" (UID: "33626a56-ce7d-49b8-9f98-316044f8fa80"). InnerVolumeSpecName "kube-api-access-j9vml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:11.483947 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.483929    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33626a56-ce7d-49b8-9f98-316044f8fa80-plugins-config-volume" pod "33626a56-ce7d-49b8-9f98-316044f8fa80" (UID: "33626a56-ce7d-49b8-9f98-316044f8fa80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:11.531398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.531366    5725 generic.go:356] "Generic (PLEG): container finished" podID="5b3b02e0-06f8-4865-8301-fc5f33a74972" containerID="1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d" exitCode=0
Nov 01 12:56:11.531599 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.531429    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"
Nov 01 12:56:11.531599 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.531450    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc" event={"ID":"5b3b02e0-06f8-4865-8301-fc5f33a74972","Type":"ContainerDied","Data":"1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d"}
Nov 01 12:56:11.531599 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.531506    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc" event={"ID":"5b3b02e0-06f8-4865-8301-fc5f33a74972","Type":"ContainerDied","Data":"21705c99e8724c3b27a1041df2feb7c9400ef67bd33145626d1a1b63e90cc491"}
Nov 01 12:56:11.531599 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.531539    5725 scope.go:122] "RemoveContainer" containerID="1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d"
Nov 01 12:56:11.533373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.533342    5725 generic.go:356] "Generic (PLEG): container finished" podID="33626a56-ce7d-49b8-9f98-316044f8fa80" containerID="d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a" exitCode=0
Nov 01 12:56:11.533493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.533407    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg" event={"ID":"33626a56-ce7d-49b8-9f98-316044f8fa80","Type":"ContainerDied","Data":"d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a"}
Nov 01 12:56:11.533493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.533452    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg" event={"ID":"33626a56-ce7d-49b8-9f98-316044f8fa80","Type":"ContainerDied","Data":"2968adaa0c57996e63ee4597068751c661c714ae26c831e382b8b666b02c8d67"}
Nov 01 12:56:11.533493 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.533413    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"
Nov 01 12:56:11.544884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.544863    5725 scope.go:122] "RemoveContainer" containerID="1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d"
Nov 01 12:56:11.545237 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:56:11.545215    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d\": container with ID starting with 1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d not found: ID does not exist" containerID="1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d"
Nov 01 12:56:11.545341 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.545245    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d"} err="failed to get container status \"1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d\": rpc error: code = NotFound desc = could not find container \"1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d\": container with ID starting with 1af0a31125a0a39c5bd9af941fda970db9cc6ad164bdb7135a1a72b55783090d not found: ID does not exist"
Nov 01 12:56:11.545341 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.545265    5725 scope.go:122] "RemoveContainer" containerID="d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a"
Nov 01 12:56:11.555443 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.555422    5725 scope.go:122] "RemoveContainer" containerID="d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a"
Nov 01 12:56:11.555791 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:56:11.555764    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a\": container with ID starting with d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a not found: ID does not exist" containerID="d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a"
Nov 01 12:56:11.555871 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.555798    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a"} err="failed to get container status \"d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a\": rpc error: code = NotFound desc = could not find container \"d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a\": container with ID starting with d107bbe8fb7db2fcd2d494841e40377fce981a6dea0d77427576f3fc38524d5a not found: ID does not exist"
Nov 01 12:56:11.582215 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.582191    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33626a56-ce7d-49b8-9f98-316044f8fa80-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:11.582215 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.582215    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9vml\" (UniqueName: \"kubernetes.io/projected/33626a56-ce7d-49b8-9f98-316044f8fa80-kube-api-access-j9vml\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:11.688708 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.688516    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"]
Nov 01 12:56:11.695460 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.695426    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f893c2-544bff7bc7-mcmzc"]
Nov 01 12:56:11.798692 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.798613    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"]
Nov 01 12:56:11.805557 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:11.805531    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f893c2-978c6d8f9-l2nfg"]
Nov 01 12:56:12.617323 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:12.617289    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33626a56-ce7d-49b8-9f98-316044f8fa80" path="/var/lib/kubelet/pods/33626a56-ce7d-49b8-9f98-316044f8fa80/volumes"
Nov 01 12:56:12.617748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:12.617732    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3b02e0-06f8-4865-8301-fc5f33a74972" path="/var/lib/kubelet/pods/5b3b02e0-06f8-4865-8301-fc5f33a74972/volumes"
Nov 01 12:56:22.726527 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:22.726353    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"]
Nov 01 12:56:22.923976 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:22.923938    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"]
Nov 01 12:56:27.728355 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:27.728290    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6" podUID="57b9baf9-fd4a-4f71-97d1-6d2b452a55fc" containerName="payload-processing" containerID="cri-o://41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d" gracePeriod=25
Nov 01 12:56:27.924461 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:27.924204    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr" podUID="550d565d-e086-4087-9777-9ababfbbd3e2" containerName="payload-pre-processing" containerID="cri-o://ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b" gracePeriod=25
Nov 01 12:56:28.060108 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.060075    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:56:28.203777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.203751    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:56:28.235404 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.235371    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-kube-api-access-pqs62\" (UniqueName: \"kubernetes.io/projected/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-kube-api-access-pqs62\") pod \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\" (UID: \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\") "
Nov 01 12:56:28.235595 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.235528    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-plugins-config-volume\") pod \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\" (UID: \"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc\") "
Nov 01 12:56:28.236179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.236147    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-plugins-config-volume" pod "57b9baf9-fd4a-4f71-97d1-6d2b452a55fc" (UID: "57b9baf9-fd4a-4f71-97d1-6d2b452a55fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:28.238177 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.238147    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-kube-api-access-pqs62" pod "57b9baf9-fd4a-4f71-97d1-6d2b452a55fc" (UID: "57b9baf9-fd4a-4f71-97d1-6d2b452a55fc"). InnerVolumeSpecName "kube-api-access-pqs62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:28.337279 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.337191    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550d565d-e086-4087-9777-9ababfbbd3e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550d565d-e086-4087-9777-9ababfbbd3e2-plugins-config-volume\") pod \"550d565d-e086-4087-9777-9ababfbbd3e2\" (UID: \"550d565d-e086-4087-9777-9ababfbbd3e2\") "
Nov 01 12:56:28.337644 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.337617    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550d565d-e086-4087-9777-9ababfbbd3e2-kube-api-access-nw2vf\" (UniqueName: \"kubernetes.io/projected/550d565d-e086-4087-9777-9ababfbbd3e2-kube-api-access-nw2vf\") pod \"550d565d-e086-4087-9777-9ababfbbd3e2\" (UID: \"550d565d-e086-4087-9777-9ababfbbd3e2\") "
Nov 01 12:56:28.337980 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.337613    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550d565d-e086-4087-9777-9ababfbbd3e2-plugins-config-volume" pod "550d565d-e086-4087-9777-9ababfbbd3e2" (UID: "550d565d-e086-4087-9777-9ababfbbd3e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:28.338422 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.338397    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqs62\" (UniqueName: \"kubernetes.io/projected/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-kube-api-access-pqs62\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:28.338528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.338428    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:28.338528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.338443    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550d565d-e086-4087-9777-9ababfbbd3e2-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:28.340416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.340393    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550d565d-e086-4087-9777-9ababfbbd3e2-kube-api-access-nw2vf" pod "550d565d-e086-4087-9777-9ababfbbd3e2" (UID: "550d565d-e086-4087-9777-9ababfbbd3e2"). InnerVolumeSpecName "kube-api-access-nw2vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:28.439462 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.439427    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw2vf\" (UniqueName: \"kubernetes.io/projected/550d565d-e086-4087-9777-9ababfbbd3e2-kube-api-access-nw2vf\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:28.679244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.679208    5725 generic.go:356] "Generic (PLEG): container finished" podID="57b9baf9-fd4a-4f71-97d1-6d2b452a55fc" containerID="41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d" exitCode=0
Nov 01 12:56:28.679429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.679271    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"
Nov 01 12:56:28.679429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.679289    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6" event={"ID":"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc","Type":"ContainerDied","Data":"41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d"}
Nov 01 12:56:28.679429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.679351    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6" event={"ID":"57b9baf9-fd4a-4f71-97d1-6d2b452a55fc","Type":"ContainerDied","Data":"cf9483325cee643800f025a03003cb69144f9185f41f1a8329d9bb99479fc940"}
Nov 01 12:56:28.679429 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.679384    5725 scope.go:122] "RemoveContainer" containerID="41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d"
Nov 01 12:56:28.680992 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.680966    5725 generic.go:356] "Generic (PLEG): container finished" podID="550d565d-e086-4087-9777-9ababfbbd3e2" containerID="ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b" exitCode=0
Nov 01 12:56:28.681170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.681015    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"
Nov 01 12:56:28.681170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.681080    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr" event={"ID":"550d565d-e086-4087-9777-9ababfbbd3e2","Type":"ContainerDied","Data":"ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b"}
Nov 01 12:56:28.681170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.681131    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr" event={"ID":"550d565d-e086-4087-9777-9ababfbbd3e2","Type":"ContainerDied","Data":"c2540ebf55fa934720a7297ab6b82120ce159f00c1cbe916aefa1bef413a22d9"}
Nov 01 12:56:28.690989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.690972    5725 scope.go:122] "RemoveContainer" containerID="41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d"
Nov 01 12:56:28.691342 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:56:28.691309    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d\": container with ID starting with 41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d not found: ID does not exist" containerID="41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d"
Nov 01 12:56:28.691453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.691350    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d"} err="failed to get container status \"41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d\": rpc error: code = NotFound desc = could not find container \"41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d\": container with ID starting with 41d17614d0dd8f2c06cd63bc2ccb8eb52c858087d8597ae5035b6560eafac27d not found: ID does not exist"
Nov 01 12:56:28.691453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.691374    5725 scope.go:122] "RemoveContainer" containerID="ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b"
Nov 01 12:56:28.702836 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.702814    5725 scope.go:122] "RemoveContainer" containerID="ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b"
Nov 01 12:56:28.703123 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:56:28.703105    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b\": container with ID starting with ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b not found: ID does not exist" containerID="ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b"
Nov 01 12:56:28.703193 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.703132    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b"} err="failed to get container status \"ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b\": rpc error: code = NotFound desc = could not find container \"ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b\": container with ID starting with ade80baa8351e96d29d6216f72652dd099b084c5823467bbd5d0999dab6c8f6b not found: ID does not exist"
Nov 01 12:56:28.838891 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.838857    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"]
Nov 01 12:56:28.883442 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.883361    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c4bbec-74546df69f-w7ssr"]
Nov 01 12:56:28.942851 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.942633    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"]
Nov 01 12:56:28.948761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:28.948727    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c4bbec-7dfdf68bcc-qqcl6"]
Nov 01 12:56:30.617441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:30.617409    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550d565d-e086-4087-9777-9ababfbbd3e2" path="/var/lib/kubelet/pods/550d565d-e086-4087-9777-9ababfbbd3e2/volumes"
Nov 01 12:56:30.617856 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:30.617842    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b9baf9-fd4a-4f71-97d1-6d2b452a55fc" path="/var/lib/kubelet/pods/57b9baf9-fd4a-4f71-97d1-6d2b452a55fc/volumes"
Nov 01 12:56:31.028213 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:56:31.028179    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:56:38.039905 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.039844    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"]
Nov 01 12:56:38.040961 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.040081    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns" podUID="e53102bf-61e4-44c5-a4a1-a15ed898a623" containerName="maas-api" containerID="cri-o://b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66" gracePeriod=30
Nov 01 12:56:38.435676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.435647    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:56:38.486641 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.486605    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53102bf-61e4-44c5-a4a1-a15ed898a623-kube-api-access-rtzx5\" (UniqueName: \"kubernetes.io/projected/e53102bf-61e4-44c5-a4a1-a15ed898a623-kube-api-access-rtzx5\") pod \"e53102bf-61e4-44c5-a4a1-a15ed898a623\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") "
Nov 01 12:56:38.486834 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.486684    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e53102bf-61e4-44c5-a4a1-a15ed898a623-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53102bf-61e4-44c5-a4a1-a15ed898a623-openshift-service-ca\") pod \"e53102bf-61e4-44c5-a4a1-a15ed898a623\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") "
Nov 01 12:56:38.486834 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.486727    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-metrics-certs\") pod \"e53102bf-61e4-44c5-a4a1-a15ed898a623\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") "
Nov 01 12:56:38.486965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.486851    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls\") pod \"e53102bf-61e4-44c5-a4a1-a15ed898a623\" (UID: \"e53102bf-61e4-44c5-a4a1-a15ed898a623\") "
Nov 01 12:56:38.487104 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.487078    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53102bf-61e4-44c5-a4a1-a15ed898a623-openshift-service-ca" pod "e53102bf-61e4-44c5-a4a1-a15ed898a623" (UID: "e53102bf-61e4-44c5-a4a1-a15ed898a623"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:38.487388 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.487360    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53102bf-61e4-44c5-a4a1-a15ed898a623-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:38.489520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.489492    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53102bf-61e4-44c5-a4a1-a15ed898a623-kube-api-access-rtzx5" pod "e53102bf-61e4-44c5-a4a1-a15ed898a623" (UID: "e53102bf-61e4-44c5-a4a1-a15ed898a623"). InnerVolumeSpecName "kube-api-access-rtzx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:38.490148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.490115    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls" pod "e53102bf-61e4-44c5-a4a1-a15ed898a623" (UID: "e53102bf-61e4-44c5-a4a1-a15ed898a623"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:56:38.490148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.490138    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-metrics-certs" pod "e53102bf-61e4-44c5-a4a1-a15ed898a623" (UID: "e53102bf-61e4-44c5-a4a1-a15ed898a623"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:56:38.600114 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.596200    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtzx5\" (UniqueName: \"kubernetes.io/projected/e53102bf-61e4-44c5-a4a1-a15ed898a623-kube-api-access-rtzx5\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:38.600114 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.596252    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:38.600114 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.596267    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53102bf-61e4-44c5-a4a1-a15ed898a623-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:38.774153 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.774102    5725 generic.go:356] "Generic (PLEG): container finished" podID="e53102bf-61e4-44c5-a4a1-a15ed898a623" containerID="b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66" exitCode=0
Nov 01 12:56:38.774424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.774157    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"
Nov 01 12:56:38.774424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.774187    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns" event={"ID":"e53102bf-61e4-44c5-a4a1-a15ed898a623","Type":"ContainerDied","Data":"b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66"}
Nov 01 12:56:38.774424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.774266    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns" event={"ID":"e53102bf-61e4-44c5-a4a1-a15ed898a623","Type":"ContainerDied","Data":"e35446dbee67c6e40ce358f0aa557dc78ffd251c956ef3eef3f5e5d752b8ebab"}
Nov 01 12:56:38.774424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.774307    5725 scope.go:122] "RemoveContainer" containerID="b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66"
Nov 01 12:56:38.792578 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.792549    5725 scope.go:122] "RemoveContainer" containerID="b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66"
Nov 01 12:56:38.795578 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:56:38.795547    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66\": container with ID starting with b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66 not found: ID does not exist" containerID="b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66"
Nov 01 12:56:38.795706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.795588    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66"} err="failed to get container status \"b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66\": rpc error: code = NotFound desc = could not find container \"b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66\": container with ID starting with b5e795bc8d68b2d632eb750132d222345e4e34a81de23232290ad6a572e38f66 not found: ID does not exist"
Nov 01 12:56:38.933739 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.933644    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"]
Nov 01 12:56:38.980595 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:38.980555    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c4bbec-c9fd9d856-fqwns"]
Nov 01 12:56:40.128409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.128366    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"]
Nov 01 12:56:40.129019 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129001    5725 cpu_manager.go:414] "containerMap: removing container" podUID="57b9baf9-fd4a-4f71-97d1-6d2b452a55fc" containerName="payload-processing"
Nov 01 12:56:40.129019 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129019    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b9baf9-fd4a-4f71-97d1-6d2b452a55fc" containerName="payload-processing"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129030    5725 cpu_manager.go:414] "containerMap: removing container" podUID="1bb5d696-4939-471e-8ab9-1cdb2d9cd528" containerName="test-curl-7091-162683"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129035    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb5d696-4939-471e-8ab9-1cdb2d9cd528" containerName="test-curl-7091-162683"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129063    5725 cpu_manager.go:414] "containerMap: removing container" podUID="33626a56-ce7d-49b8-9f98-316044f8fa80" containerName="payload-pre-processing"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129068    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33626a56-ce7d-49b8-9f98-316044f8fa80" containerName="payload-pre-processing"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129086    5725 cpu_manager.go:414] "containerMap: removing container" podUID="5b3b02e0-06f8-4865-8301-fc5f33a74972" containerName="payload-processing"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129090    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3b02e0-06f8-4865-8301-fc5f33a74972" containerName="payload-processing"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129113    5725 cpu_manager.go:414] "containerMap: removing container" podUID="e53102bf-61e4-44c5-a4a1-a15ed898a623" containerName="maas-api"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129118    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53102bf-61e4-44c5-a4a1-a15ed898a623" containerName="maas-api"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129127    5725 cpu_manager.go:414] "containerMap: removing container" podUID="550d565d-e086-4087-9777-9ababfbbd3e2" containerName="payload-pre-processing"
Nov 01 12:56:40.129181 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129132    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550d565d-e086-4087-9777-9ababfbbd3e2" containerName="payload-pre-processing"
Nov 01 12:56:40.129700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129229    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53102bf-61e4-44c5-a4a1-a15ed898a623" containerName="maas-api"
Nov 01 12:56:40.129700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129240    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="550d565d-e086-4087-9777-9ababfbbd3e2" containerName="payload-pre-processing"
Nov 01 12:56:40.129700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129250    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b9baf9-fd4a-4f71-97d1-6d2b452a55fc" containerName="payload-processing"
Nov 01 12:56:40.129700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129256    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb5d696-4939-471e-8ab9-1cdb2d9cd528" containerName="test-curl-7091-162683"
Nov 01 12:56:40.129700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129262    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="33626a56-ce7d-49b8-9f98-316044f8fa80" containerName="payload-pre-processing"
Nov 01 12:56:40.129700 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.129270    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3b02e0-06f8-4865-8301-fc5f33a74972" containerName="payload-processing"
Nov 01 12:56:40.134005 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.133984    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.215786 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.215748    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.215981 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.215907    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d957120b-c603-4caf-a992-b478e28008c5-openshift-service-ca\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.216101 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.216082    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqfs9\" (UniqueName: \"kubernetes.io/projected/d957120b-c603-4caf-a992-b478e28008c5-kube-api-access-dqfs9\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.216206 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.216188    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-metrics-certs\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.232777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.232744    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2810c2-serving-cert\""
Nov 01 12:56:40.238258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.238232    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"]
Nov 01 12:56:40.317732 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.317702    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.317927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.317833    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d957120b-c603-4caf-a992-b478e28008c5-openshift-service-ca\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.317927 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:56:40.317848    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-serving-cert: secret "maas-api-e2e-api-b-2810c2-serving-cert" not found
Nov 01 12:56:40.317927 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:56:40.317910    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls podName:d957120b-c603-4caf-a992-b478e28008c5 nodeName:}" failed. No retries permitted until 2026-11-01 12:56:40.817889378 +0000 UTC m=+2042.325080592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls") pod "maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb" (UID: "d957120b-c603-4caf-a992-b478e28008c5") : secret "maas-api-e2e-api-b-2810c2-serving-cert" not found
Nov 01 12:56:40.318141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.317980    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfs9\" (UniqueName: \"kubernetes.io/projected/d957120b-c603-4caf-a992-b478e28008c5-kube-api-access-dqfs9\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.318141 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.318062    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-metrics-certs\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.318507 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.318489    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d957120b-c603-4caf-a992-b478e28008c5-openshift-service-ca\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.320617 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.320588    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-metrics-certs\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.375529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.375495    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqfs9\" (UniqueName: \"kubernetes.io/projected/d957120b-c603-4caf-a992-b478e28008c5-kube-api-access-dqfs9\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.619830 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.619798    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53102bf-61e4-44c5-a4a1-a15ed898a623" path="/var/lib/kubelet/pods/e53102bf-61e4-44c5-a4a1-a15ed898a623/volumes"
Nov 01 12:56:40.825553 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.825519    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:40.828223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:40.828195    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls\") pod \"maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:41.048145 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:41.048113    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:41.426164 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:41.426084    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"]
Nov 01 12:56:41.432291 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:56:41.432251    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd957120b_c603_4caf_a992_b478e28008c5.slice/crio-701a2f5a76237b453504651088ee5c17c3cf7ac4a66a900d68143db90b04baed WatchSource:0}: Error finding container 701a2f5a76237b453504651088ee5c17c3cf7ac4a66a900d68143db90b04baed: Status 404 returned error can't find the container with id 701a2f5a76237b453504651088ee5c17c3cf7ac4a66a900d68143db90b04baed
Nov 01 12:56:41.433748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:41.433725    5725 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:56:41.513552 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:41.513521    5725 scope.go:122] "RemoveContainer" containerID="f247258b818517d5f4b3875e436b4d5aecb0fc20bac16626af8cd72601c7340b"
Nov 01 12:56:41.524295 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:41.524270    5725 scope.go:122] "RemoveContainer" containerID="6967e82f7571d7cb9a88702398e51f971b2e2f173e8207ea4baf0a846a6a00f1"
Nov 01 12:56:41.801061 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:41.801009    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb" event={"ID":"d957120b-c603-4caf-a992-b478e28008c5","Type":"ContainerStarted","Data":"701a2f5a76237b453504651088ee5c17c3cf7ac4a66a900d68143db90b04baed"}
Nov 01 12:56:42.821301 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:42.813778    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb" event={"ID":"d957120b-c603-4caf-a992-b478e28008c5","Type":"ContainerStarted","Data":"bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea"}
Nov 01 12:56:42.821301 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:42.814880    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:56:42.957425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:42.957374    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb" podStartSLOduration=2.030006646 podStartE2EDuration="2.957358912s" totalImagesPullingTime="927.352266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:56:42.955116045 +0000 UTC m=+2044.462307262" watchObservedRunningTime="2026-11-01 12:56:42.957358912 +0000 UTC m=+2044.464550129"
Nov 01 12:56:49.838439 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:56:49.838407    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 12:57:01.035086 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:57:01.034020    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:57:02.010837 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.010789    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"]
Nov 01 12:57:02.011453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.011418    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j" podUID="908e5730-2323-4fbc-a1b2-f408979ee787" containerName="discovery" containerID="cri-o://7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4" gracePeriod=30
Nov 01 12:57:02.367074 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.367028    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:57:02.438846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.438811    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-cacerts\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-cacerts\") pod \"908e5730-2323-4fbc-a1b2-f408979ee787\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") "
Nov 01 12:57:02.439080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.438897    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-kube-api-access-nmb97\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-kube-api-access-nmb97\") pod \"908e5730-2323-4fbc-a1b2-f408979ee787\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") "
Nov 01 12:57:02.439157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.439091    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908e5730-2323-4fbc-a1b2-f408979ee787-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908e5730-2323-4fbc-a1b2-f408979ee787-local-certs\") pod \"908e5730-2323-4fbc-a1b2-f408979ee787\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") "
Nov 01 12:57:02.439216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.439155    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-ca-configmap\") pod \"908e5730-2323-4fbc-a1b2-f408979ee787\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") "
Nov 01 12:57:02.439216 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.439210    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-kubeconfig\") pod \"908e5730-2323-4fbc-a1b2-f408979ee787\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") "
Nov 01 12:57:02.439373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.439247    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-dns-cert\") pod \"908e5730-2323-4fbc-a1b2-f408979ee787\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") "
Nov 01 12:57:02.439373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.439304    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-istio-token\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-istio-token\") pod \"908e5730-2323-4fbc-a1b2-f408979ee787\" (UID: \"908e5730-2323-4fbc-a1b2-f408979ee787\") "
Nov 01 12:57:02.440582 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.439993    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-ca-configmap" pod "908e5730-2323-4fbc-a1b2-f408979ee787" (UID: "908e5730-2323-4fbc-a1b2-f408979ee787"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:02.443928 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.443897    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-cacerts" pod "908e5730-2323-4fbc-a1b2-f408979ee787" (UID: "908e5730-2323-4fbc-a1b2-f408979ee787"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:02.449756 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.445759    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-istio-token" pod "908e5730-2323-4fbc-a1b2-f408979ee787" (UID: "908e5730-2323-4fbc-a1b2-f408979ee787"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:02.449756 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.446094    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908e5730-2323-4fbc-a1b2-f408979ee787-local-certs" pod "908e5730-2323-4fbc-a1b2-f408979ee787" (UID: "908e5730-2323-4fbc-a1b2-f408979ee787"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:02.449756 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.448951    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-kube-api-access-nmb97" pod "908e5730-2323-4fbc-a1b2-f408979ee787" (UID: "908e5730-2323-4fbc-a1b2-f408979ee787"). InnerVolumeSpecName "kube-api-access-nmb97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:02.449756 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.449370    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-dns-cert" pod "908e5730-2323-4fbc-a1b2-f408979ee787" (UID: "908e5730-2323-4fbc-a1b2-f408979ee787"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:02.454152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.454115    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-kubeconfig" pod "908e5730-2323-4fbc-a1b2-f408979ee787" (UID: "908e5730-2323-4fbc-a1b2-f408979ee787"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:02.541655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.541623    5725 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908e5730-2323-4fbc-a1b2-f408979ee787-local-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:02.541655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.541652    5725 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-ca-configmap\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:02.541655 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.541663    5725 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-kubeconfig\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:02.541932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.541672    5725 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-istio-csr-dns-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:02.541932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.541707    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:02.541932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.541716    5725 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/908e5730-2323-4fbc-a1b2-f408979ee787-cacerts\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:02.541932 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:02.541726    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmb97\" (UniqueName: \"kubernetes.io/projected/908e5730-2323-4fbc-a1b2-f408979ee787-kube-api-access-nmb97\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:03.100515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:03.100473    5725 generic.go:356] "Generic (PLEG): container finished" podID="908e5730-2323-4fbc-a1b2-f408979ee787" containerID="7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4" exitCode=0
Nov 01 12:57:03.100819 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:03.100801    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j" event={"ID":"908e5730-2323-4fbc-a1b2-f408979ee787","Type":"ContainerDied","Data":"7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4"}
Nov 01 12:57:03.100903 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:03.100850    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j" event={"ID":"908e5730-2323-4fbc-a1b2-f408979ee787","Type":"ContainerDied","Data":"88733a2e8f33f59dc25c49f92911e10af3d9d3417a36e2fc4fb3fdb737862a08"}
Nov 01 12:57:03.100973 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:03.100927    5725 scope.go:122] "RemoveContainer" containerID="7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4"
Nov 01 12:57:03.101144 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:03.101117    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"
Nov 01 12:57:03.120109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:03.120077    5725 scope.go:122] "RemoveContainer" containerID="7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4"
Nov 01 12:57:03.120545 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:57:03.120522    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4\": container with ID starting with 7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4 not found: ID does not exist" containerID="7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4"
Nov 01 12:57:03.120613 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:03.120553    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4"} err="failed to get container status \"7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4\": rpc error: code = NotFound desc = could not find container \"7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4\": container with ID starting with 7c7c1ab96180a8222e10ee6fbff156dc52a869349c8bff0ab4a693bd134c0ea4 not found: ID does not exist"
Nov 01 12:57:03.258991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:03.258947    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"]
Nov 01 12:57:03.263499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:03.263469    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7zz2j"]
Nov 01 12:57:04.617320 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:04.617286    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908e5730-2323-4fbc-a1b2-f408979ee787" path="/var/lib/kubelet/pods/908e5730-2323-4fbc-a1b2-f408979ee787/volumes"
Nov 01 12:57:07.414076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:07.413638    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"]
Nov 01 12:57:07.414076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:07.413943    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy" containerID="cri-o://4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d" gracePeriod=30
Nov 01 12:57:08.651448 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:08.651412    5725 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:57:08.651814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:08.651475    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:57:08.843484 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:08.843443    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"]
Nov 01 12:57:08.843751 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:08.843714    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw" podUID="27d13f06-e631-463e-95ba-95ae13f3fa5e" containerName="maas-api" containerID="cri-o://f1956216e61a3197bef494b4dc8189f583dc27fa6d0ce66d09dd79ce06b5a3f8" gracePeriod=30
Nov 01 12:57:09.188408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.187252    5725 generic.go:356] "Generic (PLEG): container finished" podID="27d13f06-e631-463e-95ba-95ae13f3fa5e" containerID="f1956216e61a3197bef494b4dc8189f583dc27fa6d0ce66d09dd79ce06b5a3f8" exitCode=0
Nov 01 12:57:09.188408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.187497    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw" event={"ID":"27d13f06-e631-463e-95ba-95ae13f3fa5e","Type":"ContainerDied","Data":"f1956216e61a3197bef494b4dc8189f583dc27fa6d0ce66d09dd79ce06b5a3f8"}
Nov 01 12:57:09.213747 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.213717    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:57:09.275660 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.275624    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls\") pod \"27d13f06-e631-463e-95ba-95ae13f3fa5e\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") "
Nov 01 12:57:09.275858 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.275714    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-metrics-certs\") pod \"27d13f06-e631-463e-95ba-95ae13f3fa5e\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") "
Nov 01 12:57:09.275858 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.275764    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d13f06-e631-463e-95ba-95ae13f3fa5e-kube-api-access-r4z28\" (UniqueName: \"kubernetes.io/projected/27d13f06-e631-463e-95ba-95ae13f3fa5e-kube-api-access-r4z28\") pod \"27d13f06-e631-463e-95ba-95ae13f3fa5e\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") "
Nov 01 12:57:09.275858 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.275815    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d13f06-e631-463e-95ba-95ae13f3fa5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d13f06-e631-463e-95ba-95ae13f3fa5e-openshift-service-ca\") pod \"27d13f06-e631-463e-95ba-95ae13f3fa5e\" (UID: \"27d13f06-e631-463e-95ba-95ae13f3fa5e\") "
Nov 01 12:57:09.279080 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.277321    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d13f06-e631-463e-95ba-95ae13f3fa5e-openshift-service-ca" pod "27d13f06-e631-463e-95ba-95ae13f3fa5e" (UID: "27d13f06-e631-463e-95ba-95ae13f3fa5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:09.279626 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.279569    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-metrics-certs" pod "27d13f06-e631-463e-95ba-95ae13f3fa5e" (UID: "27d13f06-e631-463e-95ba-95ae13f3fa5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:09.280121 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.280043    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls" pod "27d13f06-e631-463e-95ba-95ae13f3fa5e" (UID: "27d13f06-e631-463e-95ba-95ae13f3fa5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:09.284573 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.284545    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d13f06-e631-463e-95ba-95ae13f3fa5e-kube-api-access-r4z28" pod "27d13f06-e631-463e-95ba-95ae13f3fa5e" (UID: "27d13f06-e631-463e-95ba-95ae13f3fa5e"). InnerVolumeSpecName "kube-api-access-r4z28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:09.385129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.383813    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"]
Nov 01 12:57:09.385129 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.384074    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy" containerID="cri-o://f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d" gracePeriod=30
Nov 01 12:57:09.386408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.386378    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:09.386600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.386583    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4z28\" (UniqueName: \"kubernetes.io/projected/27d13f06-e631-463e-95ba-95ae13f3fa5e-kube-api-access-r4z28\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:09.397136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.397105    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d13f06-e631-463e-95ba-95ae13f3fa5e-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:09.397350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:09.397335    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d13f06-e631-463e-95ba-95ae13f3fa5e-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:10.196842 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:10.196810    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"
Nov 01 12:57:10.196842 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:10.196821    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw" event={"ID":"27d13f06-e631-463e-95ba-95ae13f3fa5e","Type":"ContainerDied","Data":"42fecdded9ab8711a333516b7ae9e28b452d3adca21b259bc49b15b7aba3add3"}
Nov 01 12:57:10.197466 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:10.196923    5725 scope.go:122] "RemoveContainer" containerID="f1956216e61a3197bef494b4dc8189f583dc27fa6d0ce66d09dd79ce06b5a3f8"
Nov 01 12:57:10.353229 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:10.353195    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"]
Nov 01 12:57:10.356940 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:10.356910    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6b9030-8977f6855-vlbvw"]
Nov 01 12:57:10.622244 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:10.620296    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d13f06-e631-463e-95ba-95ae13f3fa5e" path="/var/lib/kubelet/pods/27d13f06-e631-463e-95ba-95ae13f3fa5e/volumes"
Nov 01 12:57:12.860220 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.860191    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:57:12.945756 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.945721    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-certs\") pod \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") "
Nov 01 12:57:12.945957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.945783    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-data\") pod \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") "
Nov 01 12:57:12.945957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.945832    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-kube-api-access-cq6sv\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-kube-api-access-cq6sv\") pod \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") "
Nov 01 12:57:12.945957 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.945912    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-credential-socket\") pod \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") "
Nov 01 12:57:12.946194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.945994    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-envoy\") pod \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") "
Nov 01 12:57:12.946194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946064    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-podinfo\") pod \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") "
Nov 01 12:57:12.946194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946078    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-data" pod "6db77c69-a8d9-4a5a-8b1b-955a6b534748" (UID: "6db77c69-a8d9-4a5a-8b1b-955a6b534748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.946194 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946103    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-token\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-token\") pod \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") "
Nov 01 12:57:12.946430 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946236    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-socket\") pod \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") "
Nov 01 12:57:12.946430 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946271    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istiod-ca-cert\") pod \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\" (UID: \"6db77c69-a8d9-4a5a-8b1b-955a6b534748\") "
Nov 01 12:57:12.946430 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946100    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-certs" pod "6db77c69-a8d9-4a5a-8b1b-955a6b534748" (UID: "6db77c69-a8d9-4a5a-8b1b-955a6b534748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.946430 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946210    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-credential-socket" pod "6db77c69-a8d9-4a5a-8b1b-955a6b534748" (UID: "6db77c69-a8d9-4a5a-8b1b-955a6b534748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.946674 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946641    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-socket" pod "6db77c69-a8d9-4a5a-8b1b-955a6b534748" (UID: "6db77c69-a8d9-4a5a-8b1b-955a6b534748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.947045 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946939    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.947045 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946968    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.947045 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946985    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.947045 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946999    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.947045 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.946936    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istiod-ca-cert" pod "6db77c69-a8d9-4a5a-8b1b-955a6b534748" (UID: "6db77c69-a8d9-4a5a-8b1b-955a6b534748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:12.949258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.949231    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-podinfo" pod "6db77c69-a8d9-4a5a-8b1b-955a6b534748" (UID: "6db77c69-a8d9-4a5a-8b1b-955a6b534748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:57:12.949395 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.949292    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-token" pod "6db77c69-a8d9-4a5a-8b1b-955a6b534748" (UID: "6db77c69-a8d9-4a5a-8b1b-955a6b534748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.950492 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.950465    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-kube-api-access-cq6sv" pod "6db77c69-a8d9-4a5a-8b1b-955a6b534748" (UID: "6db77c69-a8d9-4a5a-8b1b-955a6b534748"). InnerVolumeSpecName "kube-api-access-cq6sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.950617 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:12.950497    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-envoy" pod "6db77c69-a8d9-4a5a-8b1b-955a6b534748" (UID: "6db77c69-a8d9-4a5a-8b1b-955a6b534748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:13.048661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.048622    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:13.048661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.048662    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq6sv\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-kube-api-access-cq6sv\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:13.048915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.048678    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:13.048915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.048690    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:13.048915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.048703    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db77c69-a8d9-4a5a-8b1b-955a6b534748-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:13.224406 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.224377    5725 generic.go:356] "Generic (PLEG): container finished" podID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerID="4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d" exitCode=0
Nov 01 12:57:13.224598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.224425    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"
Nov 01 12:57:13.224598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.224469    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" event={"ID":"6db77c69-a8d9-4a5a-8b1b-955a6b534748","Type":"ContainerDied","Data":"4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d"}
Nov 01 12:57:13.224598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.224521    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw" event={"ID":"6db77c69-a8d9-4a5a-8b1b-955a6b534748","Type":"ContainerDied","Data":"1f41ff60349e360c68596ba5668f6db4ae91b8d2bd447f9ad7e2876ee87e29ed"}
Nov 01 12:57:13.224598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.224545    5725 scope.go:122] "RemoveContainer" containerID="4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d"
Nov 01 12:57:13.235718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.235696    5725 scope.go:122] "RemoveContainer" containerID="4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d"
Nov 01 12:57:13.236021 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:57:13.235995    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d\": container with ID starting with 4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d not found: ID does not exist" containerID="4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d"
Nov 01 12:57:13.236186 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.236030    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d"} err="failed to get container status \"4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d\": rpc error: code = NotFound desc = could not find container \"4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d\": container with ID starting with 4c7efb8abfd6a909fae00f9f47ecb5f3fb46a8c745791c5207d03d28f8798e9d not found: ID does not exist"
Nov 01 12:57:13.381945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.381909    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"]
Nov 01 12:57:13.428455 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:13.428423    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-ct2cw"]
Nov 01 12:57:14.618799 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.618766    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" path="/var/lib/kubelet/pods/6db77c69-a8d9-4a5a-8b1b-955a6b534748/volumes"
Nov 01 12:57:14.740093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.739996    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:57:14.872964 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.872927    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-token\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-token\") pod \"f3dae956-54ad-41d5-8441-f9fec077acfd\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") "
Nov 01 12:57:14.873150 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.872985    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-credential-socket\") pod \"f3dae956-54ad-41d5-8441-f9fec077acfd\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") "
Nov 01 12:57:14.873150 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873087    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3dae956-54ad-41d5-8441-f9fec077acfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dae956-54ad-41d5-8441-f9fec077acfd-istiod-ca-cert\") pod \"f3dae956-54ad-41d5-8441-f9fec077acfd\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") "
Nov 01 12:57:14.873302 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873190    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-podinfo\") pod \"f3dae956-54ad-41d5-8441-f9fec077acfd\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") "
Nov 01 12:57:14.873302 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873227    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-kube-api-access-wrz4h\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-kube-api-access-wrz4h\") pod \"f3dae956-54ad-41d5-8441-f9fec077acfd\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") "
Nov 01 12:57:14.873419 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873346    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-credential-socket" pod "f3dae956-54ad-41d5-8441-f9fec077acfd" (UID: "f3dae956-54ad-41d5-8441-f9fec077acfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:14.873419 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873404    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-envoy\") pod \"f3dae956-54ad-41d5-8441-f9fec077acfd\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") "
Nov 01 12:57:14.873531 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873515    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-socket\") pod \"f3dae956-54ad-41d5-8441-f9fec077acfd\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") "
Nov 01 12:57:14.873601 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873544    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3dae956-54ad-41d5-8441-f9fec077acfd-istiod-ca-cert" pod "f3dae956-54ad-41d5-8441-f9fec077acfd" (UID: "f3dae956-54ad-41d5-8441-f9fec077acfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:14.873601 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873564    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-certs\") pod \"f3dae956-54ad-41d5-8441-f9fec077acfd\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") "
Nov 01 12:57:14.873771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873752    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-certs" pod "f3dae956-54ad-41d5-8441-f9fec077acfd" (UID: "f3dae956-54ad-41d5-8441-f9fec077acfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:14.873866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873790    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-data\") pod \"f3dae956-54ad-41d5-8441-f9fec077acfd\" (UID: \"f3dae956-54ad-41d5-8441-f9fec077acfd\") "
Nov 01 12:57:14.873866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873805    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-socket" pod "f3dae956-54ad-41d5-8441-f9fec077acfd" (UID: "f3dae956-54ad-41d5-8441-f9fec077acfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:14.873986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.873956    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-data" pod "f3dae956-54ad-41d5-8441-f9fec077acfd" (UID: "f3dae956-54ad-41d5-8441-f9fec077acfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:14.874392 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.874368    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:14.874499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.874394    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:14.874499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.874409    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:14.874499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.874423    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:14.874499 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.874436    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3dae956-54ad-41d5-8441-f9fec077acfd-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:14.875693 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.875666    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-kube-api-access-wrz4h" pod "f3dae956-54ad-41d5-8441-f9fec077acfd" (UID: "f3dae956-54ad-41d5-8441-f9fec077acfd"). InnerVolumeSpecName "kube-api-access-wrz4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:14.875924 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.875900    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-podinfo" pod "f3dae956-54ad-41d5-8441-f9fec077acfd" (UID: "f3dae956-54ad-41d5-8441-f9fec077acfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:57:14.876168 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.876152    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-token" pod "f3dae956-54ad-41d5-8441-f9fec077acfd" (UID: "f3dae956-54ad-41d5-8441-f9fec077acfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:14.876348 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.876334    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-envoy" pod "f3dae956-54ad-41d5-8441-f9fec077acfd" (UID: "f3dae956-54ad-41d5-8441-f9fec077acfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:14.975717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.975685    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:14.975717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.975713    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrz4h\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-kube-api-access-wrz4h\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:14.975929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.975749    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:14.975929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:14.975763    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3dae956-54ad-41d5-8441-f9fec077acfd-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:15.264813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:15.264779    5725 generic.go:356] "Generic (PLEG): container finished" podID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerID="f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d" exitCode=0
Nov 01 12:57:15.265007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:15.264904    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" event={"ID":"f3dae956-54ad-41d5-8441-f9fec077acfd","Type":"ContainerDied","Data":"f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d"}
Nov 01 12:57:15.265007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:15.264981    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk" event={"ID":"f3dae956-54ad-41d5-8441-f9fec077acfd","Type":"ContainerDied","Data":"43412775b7f57a4503f89d09962633d9c87d04c36c846545011387746965e9e5"}
Nov 01 12:57:15.265190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:15.265008    5725 scope.go:122] "RemoveContainer" containerID="f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d"
Nov 01 12:57:15.265250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:15.265193    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"
Nov 01 12:57:15.281764 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:15.281735    5725 scope.go:122] "RemoveContainer" containerID="f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d"
Nov 01 12:57:15.282036 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:57:15.282018    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d\": container with ID starting with f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d not found: ID does not exist" containerID="f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d"
Nov 01 12:57:15.282136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:15.282042    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d"} err="failed to get container status \"f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d\": rpc error: code = NotFound desc = could not find container \"f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d\": container with ID starting with f72003f475c1706abea6ffe0fda82e3f5f12fdd7f5725fe11929c733ba31096d not found: ID does not exist"
Nov 01 12:57:15.311945 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:57:15.311920    5725 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dae956_54ad_41d5_8441_f9fec077acfd.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 12:57:15.426071 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:15.426015    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"]
Nov 01 12:57:15.473007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:15.472971    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-cmvjk"]
Nov 01 12:57:16.617598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:16.617557    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" path="/var/lib/kubelet/pods/f3dae956-54ad-41d5-8441-f9fec077acfd/volumes"
Nov 01 12:57:22.305762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:22.305672    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"]
Nov 01 12:57:22.306243 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:22.305938    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy" containerID="cri-o://1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf" gracePeriod=30
Nov 01 12:57:23.502646 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:23.502594    5725 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:57:23.503191 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:23.502670    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:57:27.581797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.581726    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:57:27.681566 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681522    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-socket\") pod \"8a3a449e-1000-4e98-84f3-c06545318c5c\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") "
Nov 01 12:57:27.681742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681588    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-envoy\") pod \"8a3a449e-1000-4e98-84f3-c06545318c5c\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") "
Nov 01 12:57:27.681742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681620    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-token\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-token\") pod \"8a3a449e-1000-4e98-84f3-c06545318c5c\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") "
Nov 01 12:57:27.681742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681662    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-kube-api-access-cp2jz\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-kube-api-access-cp2jz\") pod \"8a3a449e-1000-4e98-84f3-c06545318c5c\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") "
Nov 01 12:57:27.681742 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681719    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3a449e-1000-4e98-84f3-c06545318c5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a449e-1000-4e98-84f3-c06545318c5c-istiod-ca-cert\") pod \"8a3a449e-1000-4e98-84f3-c06545318c5c\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") "
Nov 01 12:57:27.681949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681775    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-data\") pod \"8a3a449e-1000-4e98-84f3-c06545318c5c\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") "
Nov 01 12:57:27.681949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681841    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-podinfo\") pod \"8a3a449e-1000-4e98-84f3-c06545318c5c\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") "
Nov 01 12:57:27.681949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681845    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-socket" pod "8a3a449e-1000-4e98-84f3-c06545318c5c" (UID: "8a3a449e-1000-4e98-84f3-c06545318c5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.681949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681881    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-certs\") pod \"8a3a449e-1000-4e98-84f3-c06545318c5c\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") "
Nov 01 12:57:27.681949 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.681929    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-credential-socket\") pod \"8a3a449e-1000-4e98-84f3-c06545318c5c\" (UID: \"8a3a449e-1000-4e98-84f3-c06545318c5c\") "
Nov 01 12:57:27.682277 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.682096    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-data" pod "8a3a449e-1000-4e98-84f3-c06545318c5c" (UID: "8a3a449e-1000-4e98-84f3-c06545318c5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.682356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.682332    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3a449e-1000-4e98-84f3-c06545318c5c-istiod-ca-cert" pod "8a3a449e-1000-4e98-84f3-c06545318c5c" (UID: "8a3a449e-1000-4e98-84f3-c06545318c5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:27.683268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.682808    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-credential-socket" pod "8a3a449e-1000-4e98-84f3-c06545318c5c" (UID: "8a3a449e-1000-4e98-84f3-c06545318c5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.683268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.682840    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.683268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.682877    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3a449e-1000-4e98-84f3-c06545318c5c-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.683268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.682908    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.683268 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.683000    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-certs" pod "8a3a449e-1000-4e98-84f3-c06545318c5c" (UID: "8a3a449e-1000-4e98-84f3-c06545318c5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.684363 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.684334    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-token" pod "8a3a449e-1000-4e98-84f3-c06545318c5c" (UID: "8a3a449e-1000-4e98-84f3-c06545318c5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:27.684509 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.684476    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-kube-api-access-cp2jz" pod "8a3a449e-1000-4e98-84f3-c06545318c5c" (UID: "8a3a449e-1000-4e98-84f3-c06545318c5c"). InnerVolumeSpecName "kube-api-access-cp2jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:27.684697 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.684675    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-envoy" pod "8a3a449e-1000-4e98-84f3-c06545318c5c" (UID: "8a3a449e-1000-4e98-84f3-c06545318c5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.685256 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.685228    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-podinfo" pod "8a3a449e-1000-4e98-84f3-c06545318c5c" (UID: "8a3a449e-1000-4e98-84f3-c06545318c5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:57:27.784570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.784533    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.784570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.784568    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.784570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.784579    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp2jz\" (UniqueName: \"kubernetes.io/projected/8a3a449e-1000-4e98-84f3-c06545318c5c-kube-api-access-cp2jz\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.784896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.784589    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3a449e-1000-4e98-84f3-c06545318c5c-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.784896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.784598    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.784896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:27.784611    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3a449e-1000-4e98-84f3-c06545318c5c-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:28.407521 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:28.407473    5725 generic.go:356] "Generic (PLEG): container finished" podID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerID="1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf" exitCode=0
Nov 01 12:57:28.407701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:28.407526    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"
Nov 01 12:57:28.407701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:28.407551    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" event={"ID":"8a3a449e-1000-4e98-84f3-c06545318c5c","Type":"ContainerDied","Data":"1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf"}
Nov 01 12:57:28.407701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:28.407624    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw" event={"ID":"8a3a449e-1000-4e98-84f3-c06545318c5c","Type":"ContainerDied","Data":"1f397058d0574a7e3322e06b93d4f6d31db9a7268963daf9538ba6ecce3d26fd"}
Nov 01 12:57:28.407701 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:28.407658    5725 scope.go:122] "RemoveContainer" containerID="1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf"
Nov 01 12:57:28.423795 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:28.423608    5725 scope.go:122] "RemoveContainer" containerID="1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf"
Nov 01 12:57:28.423982 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:57:28.423961    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf\": container with ID starting with 1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf not found: ID does not exist" containerID="1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf"
Nov 01 12:57:28.424026 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:28.423993    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf"} err="failed to get container status \"1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf\": rpc error: code = NotFound desc = could not find container \"1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf\": container with ID starting with 1a971734e7ae26ad7b368c98ec0e7a5d9b626b102a54311a7a2550faadaedeaf not found: ID does not exist"
Nov 01 12:57:28.620717 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:28.620688    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"]
Nov 01 12:57:28.659663 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:28.659587    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-sp4qw"]
Nov 01 12:57:30.631007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:30.628764    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" path="/var/lib/kubelet/pods/8a3a449e-1000-4e98-84f3-c06545318c5c/volumes"
Nov 01 12:57:31.032650 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:57:31.032610    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:57:34.284408 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.284357    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"]
Nov 01 12:57:34.285361 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285338    5725 cpu_manager.go:414] "containerMap: removing container" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy"
Nov 01 12:57:34.285409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285364    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy"
Nov 01 12:57:34.285409 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285399    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy"
Nov 01 12:57:34.285481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285407    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy"
Nov 01 12:57:34.285481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285426    5725 cpu_manager.go:414] "containerMap: removing container" podUID="27d13f06-e631-463e-95ba-95ae13f3fa5e" containerName="maas-api"
Nov 01 12:57:34.285481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285433    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d13f06-e631-463e-95ba-95ae13f3fa5e" containerName="maas-api"
Nov 01 12:57:34.285481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285463    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy"
Nov 01 12:57:34.285481 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285470    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy"
Nov 01 12:57:34.285631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285509    5725 cpu_manager.go:414] "containerMap: removing container" podUID="908e5730-2323-4fbc-a1b2-f408979ee787" containerName="discovery"
Nov 01 12:57:34.285631 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285517    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908e5730-2323-4fbc-a1b2-f408979ee787" containerName="discovery"
Nov 01 12:57:34.285694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285644    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3a449e-1000-4e98-84f3-c06545318c5c" containerName="istio-proxy"
Nov 01 12:57:34.285694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285664    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="908e5730-2323-4fbc-a1b2-f408979ee787" containerName="discovery"
Nov 01 12:57:34.285694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285675    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db77c69-a8d9-4a5a-8b1b-955a6b534748" containerName="istio-proxy"
Nov 01 12:57:34.285694 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285687    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3dae956-54ad-41d5-8441-f9fec077acfd" containerName="istio-proxy"
Nov 01 12:57:34.285811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.285698    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d13f06-e631-463e-95ba-95ae13f3fa5e" containerName="maas-api"
Nov 01 12:57:34.289839 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.289816    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.330548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.330513    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"]
Nov 01 12:57:34.330810 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.330750    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh" podUID="da507d04-6a1b-409f-a008-b38baf7bcd06" containerName="maas-api" containerID="cri-o://693e38d2a6bc3b3a1f0aa8f3b77e3c9f28e590da73bf88a65f1f6391b4467cb8" gracePeriod=30
Nov 01 12:57:34.342663 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.342448    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:57:34.342663 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.342512    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:57:34.359706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.359191    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec606b5-c8s4j\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.359706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.359258    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqz48\" (UniqueName: \"kubernetes.io/projected/279d8e5f-b758-42fa-acb6-af05ed4f382c-kube-api-access-rqz48\") pod \"maas-api-revoke-sub-8ec606b5-c8s4j\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.359706 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.359284    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec606b5-c8s4j\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.362285 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.362260    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"]
Nov 01 12:57:34.460382 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.460330    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec606b5-c8s4j\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.460382 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.460370    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqz48\" (UniqueName: \"kubernetes.io/projected/279d8e5f-b758-42fa-acb6-af05ed4f382c-kube-api-access-rqz48\") pod \"maas-api-revoke-sub-8ec606b5-c8s4j\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.460594 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.460397    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec606b5-c8s4j\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.461213 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.461188    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec606b5-c8s4j\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.464477 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.464426    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec606b5-c8s4j\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.466999 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.466976    5725 generic.go:356] "Generic (PLEG): container finished" podID="da507d04-6a1b-409f-a008-b38baf7bcd06" containerID="693e38d2a6bc3b3a1f0aa8f3b77e3c9f28e590da73bf88a65f1f6391b4467cb8" exitCode=0
Nov 01 12:57:34.467138 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.467097    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh" event={"ID":"da507d04-6a1b-409f-a008-b38baf7bcd06","Type":"ContainerDied","Data":"693e38d2a6bc3b3a1f0aa8f3b77e3c9f28e590da73bf88a65f1f6391b4467cb8"}
Nov 01 12:57:34.520379 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.520217    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqz48\" (UniqueName: \"kubernetes.io/projected/279d8e5f-b758-42fa-acb6-af05ed4f382c-kube-api-access-rqz48\") pod \"maas-api-revoke-sub-8ec606b5-c8s4j\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.604748 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.604672    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:34.685912 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.675829    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:57:34.784072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.776419    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da507d04-6a1b-409f-a008-b38baf7bcd06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da507d04-6a1b-409f-a008-b38baf7bcd06-openshift-service-ca\") pod \"da507d04-6a1b-409f-a008-b38baf7bcd06\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") "
Nov 01 12:57:34.784072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.776817    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da507d04-6a1b-409f-a008-b38baf7bcd06-openshift-service-ca" pod "da507d04-6a1b-409f-a008-b38baf7bcd06" (UID: "da507d04-6a1b-409f-a008-b38baf7bcd06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:34.784072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.777376    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls\") pod \"da507d04-6a1b-409f-a008-b38baf7bcd06\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") "
Nov 01 12:57:34.784072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.777469    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da507d04-6a1b-409f-a008-b38baf7bcd06-kube-api-access-dfqzq\" (UniqueName: \"kubernetes.io/projected/da507d04-6a1b-409f-a008-b38baf7bcd06-kube-api-access-dfqzq\") pod \"da507d04-6a1b-409f-a008-b38baf7bcd06\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") "
Nov 01 12:57:34.784072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.777496    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-metrics-certs\") pod \"da507d04-6a1b-409f-a008-b38baf7bcd06\" (UID: \"da507d04-6a1b-409f-a008-b38baf7bcd06\") "
Nov 01 12:57:34.784072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.778600    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da507d04-6a1b-409f-a008-b38baf7bcd06-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:34.784072 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.782727    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-metrics-certs" pod "da507d04-6a1b-409f-a008-b38baf7bcd06" (UID: "da507d04-6a1b-409f-a008-b38baf7bcd06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:34.785974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.785206    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls" pod "da507d04-6a1b-409f-a008-b38baf7bcd06" (UID: "da507d04-6a1b-409f-a008-b38baf7bcd06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:34.791420 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.786895    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da507d04-6a1b-409f-a008-b38baf7bcd06-kube-api-access-dfqzq" pod "da507d04-6a1b-409f-a008-b38baf7bcd06" (UID: "da507d04-6a1b-409f-a008-b38baf7bcd06"). InnerVolumeSpecName "kube-api-access-dfqzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:34.880298 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.880213    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:34.880298 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.880253    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqzq\" (UniqueName: \"kubernetes.io/projected/da507d04-6a1b-409f-a008-b38baf7bcd06-kube-api-access-dfqzq\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:34.880298 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:34.880269    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da507d04-6a1b-409f-a008-b38baf7bcd06-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:35.018965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:35.018925    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"]
Nov 01 12:57:35.025996 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:57:35.025953    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279d8e5f_b758_42fa_acb6_af05ed4f382c.slice/crio-4d9db2b78058c5c26b64c22c803dcba4f117d0c530fcdee01f247aad09a57f4d WatchSource:0}: Error finding container 4d9db2b78058c5c26b64c22c803dcba4f117d0c530fcdee01f247aad09a57f4d: Status 404 returned error can't find the container with id 4d9db2b78058c5c26b64c22c803dcba4f117d0c530fcdee01f247aad09a57f4d
Nov 01 12:57:35.551488 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:35.548665    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j" event={"ID":"279d8e5f-b758-42fa-acb6-af05ed4f382c","Type":"ContainerStarted","Data":"4d9db2b78058c5c26b64c22c803dcba4f117d0c530fcdee01f247aad09a57f4d"}
Nov 01 12:57:35.551986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:35.551562    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh" event={"ID":"da507d04-6a1b-409f-a008-b38baf7bcd06","Type":"ContainerDied","Data":"ae0d50d1c567e5e8bcf84978b818fd36e65345c05cfd86d8b9b25383bf9ef2e7"}
Nov 01 12:57:35.551986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:35.551651    5725 scope.go:122] "RemoveContainer" containerID="693e38d2a6bc3b3a1f0aa8f3b77e3c9f28e590da73bf88a65f1f6391b4467cb8"
Nov 01 12:57:35.551986 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:35.551890    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"
Nov 01 12:57:35.732769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:35.732736    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"]
Nov 01 12:57:35.750925 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:35.750895    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f893c2-6844566554-52qrh"]
Nov 01 12:57:36.560989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:36.560954    5725 generic.go:356] "Generic (PLEG): container finished" podID="279d8e5f-b758-42fa-acb6-af05ed4f382c" containerID="4e5b33bf3fd0638a5b2a8751bc4b3fa49a9cf93874320942e028403fc7dc15e0" exitCode=0
Nov 01 12:57:36.561465 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:36.561044    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j" event={"ID":"279d8e5f-b758-42fa-acb6-af05ed4f382c","Type":"ContainerDied","Data":"4e5b33bf3fd0638a5b2a8751bc4b3fa49a9cf93874320942e028403fc7dc15e0"}
Nov 01 12:57:36.616804 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:36.616772    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da507d04-6a1b-409f-a008-b38baf7bcd06" path="/var/lib/kubelet/pods/da507d04-6a1b-409f-a008-b38baf7bcd06/volumes"
Nov 01 12:57:37.743998 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.743973    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:37.845680 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.845601    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-openshift-ca\") pod \"279d8e5f-b758-42fa-acb6-af05ed4f382c\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") "
Nov 01 12:57:37.845861 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.845725    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-xks-ca\") pod \"279d8e5f-b758-42fa-acb6-af05ed4f382c\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") "
Nov 01 12:57:37.845861 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.845752    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279d8e5f-b758-42fa-acb6-af05ed4f382c-kube-api-access-rqz48\" (UniqueName: \"kubernetes.io/projected/279d8e5f-b758-42fa-acb6-af05ed4f382c-kube-api-access-rqz48\") pod \"279d8e5f-b758-42fa-acb6-af05ed4f382c\" (UID: \"279d8e5f-b758-42fa-acb6-af05ed4f382c\") "
Nov 01 12:57:37.846038 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.846011    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-openshift-ca" pod "279d8e5f-b758-42fa-acb6-af05ed4f382c" (UID: "279d8e5f-b758-42fa-acb6-af05ed4f382c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:37.847870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.847838    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279d8e5f-b758-42fa-acb6-af05ed4f382c-kube-api-access-rqz48" pod "279d8e5f-b758-42fa-acb6-af05ed4f382c" (UID: "279d8e5f-b758-42fa-acb6-af05ed4f382c"). InnerVolumeSpecName "kube-api-access-rqz48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:37.847994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.847942    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-xks-ca" pod "279d8e5f-b758-42fa-acb6-af05ed4f382c" (UID: "279d8e5f-b758-42fa-acb6-af05ed4f382c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:37.947510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.947478    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:37.947510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.947505    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqz48\" (UniqueName: \"kubernetes.io/projected/279d8e5f-b758-42fa-acb6-af05ed4f382c-kube-api-access-rqz48\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:37.947510 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:37.947517    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279d8e5f-b758-42fa-acb6-af05ed4f382c-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:38.579183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:38.579156    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"
Nov 01 12:57:38.579341 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:38.579190    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j" event={"ID":"279d8e5f-b758-42fa-acb6-af05ed4f382c","Type":"ContainerDied","Data":"4d9db2b78058c5c26b64c22c803dcba4f117d0c530fcdee01f247aad09a57f4d"}
Nov 01 12:57:38.579341 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:38.579234    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d9db2b78058c5c26b64c22c803dcba4f117d0c530fcdee01f247aad09a57f4d"
Nov 01 12:57:39.098956 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:39.098923    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:57:39.107821 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:39.107794    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:57:39.121208 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:39.121187    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 12:57:39.129269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:39.129247    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 12:57:47.478884 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.478845    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"]
Nov 01 12:57:47.479680 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.479657    5725 cpu_manager.go:414] "containerMap: removing container" podUID="279d8e5f-b758-42fa-acb6-af05ed4f382c" containerName="revoke-keys"
Nov 01 12:57:47.479793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.479681    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279d8e5f-b758-42fa-acb6-af05ed4f382c" containerName="revoke-keys"
Nov 01 12:57:47.479793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.479772    5725 cpu_manager.go:414] "containerMap: removing container" podUID="da507d04-6a1b-409f-a008-b38baf7bcd06" containerName="maas-api"
Nov 01 12:57:47.479793 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.479783    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da507d04-6a1b-409f-a008-b38baf7bcd06" containerName="maas-api"
Nov 01 12:57:47.479965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.479951    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="da507d04-6a1b-409f-a008-b38baf7bcd06" containerName="maas-api"
Nov 01 12:57:47.480031 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.479973    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="279d8e5f-b758-42fa-acb6-af05ed4f382c" containerName="revoke-keys"
Nov 01 12:57:47.485384 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.485360    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.494076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.494016    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6vf\" (UniqueName: \"kubernetes.io/projected/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-kube-api-access-9r6vf\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.494319 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.494291    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.494403 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.494332    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.535110 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.535076    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:57:47.535987 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.535921    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:57:47.539432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.539405    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"]
Nov 01 12:57:47.595350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.595312    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.595545 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.595448    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.595545 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.595512    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6vf\" (UniqueName: \"kubernetes.io/projected/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-kube-api-access-9r6vf\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.596140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.596111    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.599414 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.599388    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.650356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.650323    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6vf\" (UniqueName: \"kubernetes.io/projected/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-kube-api-access-9r6vf\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.799927 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.799844    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:47.902860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.902828    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"]
Nov 01 12:57:47.908336 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.908310    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:47.967860 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.967817    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"]
Nov 01 12:57:47.998968 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.998930    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgrz\" (UniqueName: \"kubernetes.io/projected/58425220-74da-46a5-ae9b-d62f85f6957a-kube-api-access-scgrz\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:47.999150 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.998980    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:47.999205 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:47.999190    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:48.100002 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.099913    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:48.100002 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.100001    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgrz\" (UniqueName: \"kubernetes.io/projected/58425220-74da-46a5-ae9b-d62f85f6957a-kube-api-access-scgrz\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:48.100293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.100035    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:48.100797 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.100777    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:48.102356 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.102336    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:48.154718 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.154676    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgrz\" (UniqueName: \"kubernetes.io/projected/58425220-74da-46a5-ae9b-d62f85f6957a-kube-api-access-scgrz\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:48.182183 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.182150    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"]
Nov 01 12:57:48.188134 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:57:48.188107    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80d83c8_6b81_458b_9507_ed1f6f0c4d63.slice/crio-4e4814ede41c8297e19731db5d9bfc66aaf0c32ca0a0f841601f8409f0c8715d WatchSource:0}: Error finding container 4e4814ede41c8297e19731db5d9bfc66aaf0c32ca0a0f841601f8409f0c8715d: Status 404 returned error can't find the container with id 4e4814ede41c8297e19731db5d9bfc66aaf0c32ca0a0f841601f8409f0c8715d
Nov 01 12:57:48.228145 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.228110    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:48.245787 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.245756    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"]
Nov 01 12:57:48.269398 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.269125    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.300677 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.300640    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"]
Nov 01 12:57:48.302437 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.302407    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2l8\" (UniqueName: \"kubernetes.io/projected/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-kube-api-access-hg2l8\") pod \"maas-api-revoke-sub-b51fa788-2lwc8\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.302572 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.302465    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b51fa788-2lwc8\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.302665 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.302608    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b51fa788-2lwc8\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.403335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.403249    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b51fa788-2lwc8\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.403497 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.403402    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2l8\" (UniqueName: \"kubernetes.io/projected/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-kube-api-access-hg2l8\") pod \"maas-api-revoke-sub-b51fa788-2lwc8\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.403497 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.403440    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b51fa788-2lwc8\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.404113 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.404088    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b51fa788-2lwc8\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.405705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.405674    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b51fa788-2lwc8\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.461578 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.461544    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2l8\" (UniqueName: \"kubernetes.io/projected/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-kube-api-access-hg2l8\") pod \"maas-api-revoke-sub-b51fa788-2lwc8\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.586708 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.586662    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:48.650501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.650132    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"]
Nov 01 12:57:48.689415 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.689382    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck" event={"ID":"c80d83c8-6b81-458b-9507-ed1f6f0c4d63","Type":"ContainerStarted","Data":"86b2338c6cfb8e0e2ad208f1f9d8e0c0490dd8f8bc9d628a19f781f4653d817a"}
Nov 01 12:57:48.689553 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.689434    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck" event={"ID":"c80d83c8-6b81-458b-9507-ed1f6f0c4d63","Type":"ContainerStarted","Data":"4e4814ede41c8297e19731db5d9bfc66aaf0c32ca0a0f841601f8409f0c8715d"}
Nov 01 12:57:48.799979 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.799925    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck" podStartSLOduration=1.799906725 podStartE2EDuration="1.799906725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:57:48.798082079 +0000 UTC m=+2110.305273288" watchObservedRunningTime="2026-11-01 12:57:48.799906725 +0000 UTC m=+2110.307097953"
Nov 01 12:57:48.984350 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:48.984313    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"]
Nov 01 12:57:48.990426 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:57:48.990393    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88057d9b_59f2_4e5f_8f1c_8057398cd2f9.slice/crio-637b7e0ed1268da6e591faecd3571aad0d586851d5d8f6fa7e8a773116f2ff09 WatchSource:0}: Error finding container 637b7e0ed1268da6e591faecd3571aad0d586851d5d8f6fa7e8a773116f2ff09: Status 404 returned error can't find the container with id 637b7e0ed1268da6e591faecd3571aad0d586851d5d8f6fa7e8a773116f2ff09
Nov 01 12:57:49.697888 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:49.697855    5725 generic.go:356] "Generic (PLEG): container finished" podID="c80d83c8-6b81-458b-9507-ed1f6f0c4d63" containerID="86b2338c6cfb8e0e2ad208f1f9d8e0c0490dd8f8bc9d628a19f781f4653d817a" exitCode=0
Nov 01 12:57:49.698299 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:49.697936    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck" event={"ID":"c80d83c8-6b81-458b-9507-ed1f6f0c4d63","Type":"ContainerDied","Data":"86b2338c6cfb8e0e2ad208f1f9d8e0c0490dd8f8bc9d628a19f781f4653d817a"}
Nov 01 12:57:49.700312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:49.700282    5725 generic.go:356] "Generic (PLEG): container finished" podID="58425220-74da-46a5-ae9b-d62f85f6957a" containerID="56067a4859d3c508da5477fad0afa812b5c6cb5facdc56e470f2d5249a7c8c5b" exitCode=0
Nov 01 12:57:49.700472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:49.700370    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs" event={"ID":"58425220-74da-46a5-ae9b-d62f85f6957a","Type":"ContainerDied","Data":"56067a4859d3c508da5477fad0afa812b5c6cb5facdc56e470f2d5249a7c8c5b"}
Nov 01 12:57:49.700472 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:49.700423    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs" event={"ID":"58425220-74da-46a5-ae9b-d62f85f6957a","Type":"ContainerStarted","Data":"a90562ca58173849b21d6e13196afa58547546c5ca0ba48ca910e35c184a7288"}
Nov 01 12:57:49.702123 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:49.702100    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8" event={"ID":"88057d9b-59f2-4e5f-8f1c-8057398cd2f9","Type":"ContainerStarted","Data":"ab29f46c8df58eef29c1e68f6ac1cd8f67047441b37a8f37b4538e12e9d766c4"}
Nov 01 12:57:49.702232 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:49.702140    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8" event={"ID":"88057d9b-59f2-4e5f-8f1c-8057398cd2f9","Type":"ContainerStarted","Data":"637b7e0ed1268da6e591faecd3571aad0d586851d5d8f6fa7e8a773116f2ff09"}
Nov 01 12:57:49.918175 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:49.918060    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8" podStartSLOduration=1.918029807 podStartE2EDuration="1.918029807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:57:49.916786041 +0000 UTC m=+2111.423977259" watchObservedRunningTime="2026-11-01 12:57:49.918029807 +0000 UTC m=+2111.425221024"
Nov 01 12:57:50.712893 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.712856    5725 generic.go:356] "Generic (PLEG): container finished" podID="88057d9b-59f2-4e5f-8f1c-8057398cd2f9" containerID="ab29f46c8df58eef29c1e68f6ac1cd8f67047441b37a8f37b4538e12e9d766c4" exitCode=0
Nov 01 12:57:50.713423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.712915    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8" event={"ID":"88057d9b-59f2-4e5f-8f1c-8057398cd2f9","Type":"ContainerDied","Data":"ab29f46c8df58eef29c1e68f6ac1cd8f67047441b37a8f37b4538e12e9d766c4"}
Nov 01 12:57:50.911458 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.911429    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:50.937240 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.937211    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58425220-74da-46a5-ae9b-d62f85f6957a-kube-api-access-scgrz\" (UniqueName: \"kubernetes.io/projected/58425220-74da-46a5-ae9b-d62f85f6957a-kube-api-access-scgrz\") pod \"58425220-74da-46a5-ae9b-d62f85f6957a\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") "
Nov 01 12:57:50.937397 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.937330    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-openshift-ca\") pod \"58425220-74da-46a5-ae9b-d62f85f6957a\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") "
Nov 01 12:57:50.937397 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.937382    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-xks-ca\") pod \"58425220-74da-46a5-ae9b-d62f85f6957a\" (UID: \"58425220-74da-46a5-ae9b-d62f85f6957a\") "
Nov 01 12:57:50.937892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.937861    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-openshift-ca" pod "58425220-74da-46a5-ae9b-d62f85f6957a" (UID: "58425220-74da-46a5-ae9b-d62f85f6957a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:50.939528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.939503    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58425220-74da-46a5-ae9b-d62f85f6957a-kube-api-access-scgrz" pod "58425220-74da-46a5-ae9b-d62f85f6957a" (UID: "58425220-74da-46a5-ae9b-d62f85f6957a"). InnerVolumeSpecName "kube-api-access-scgrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:50.939835 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.939813    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-xks-ca" pod "58425220-74da-46a5-ae9b-d62f85f6957a" (UID: "58425220-74da-46a5-ae9b-d62f85f6957a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:50.962843 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:50.962820    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:51.038518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.038486    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-openshift-ca\") pod \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") "
Nov 01 12:57:51.038709 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.038526    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-xks-ca\") pod \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") "
Nov 01 12:57:51.038709 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.038588    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-kube-api-access-9r6vf\" (UniqueName: \"kubernetes.io/projected/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-kube-api-access-9r6vf\") pod \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\" (UID: \"c80d83c8-6b81-458b-9507-ed1f6f0c4d63\") "
Nov 01 12:57:51.039212 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.039175    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-openshift-ca" pod "c80d83c8-6b81-458b-9507-ed1f6f0c4d63" (UID: "c80d83c8-6b81-458b-9507-ed1f6f0c4d63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:51.039362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.039252    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:51.039362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.039275    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58425220-74da-46a5-ae9b-d62f85f6957a-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:51.039362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.039290    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scgrz\" (UniqueName: \"kubernetes.io/projected/58425220-74da-46a5-ae9b-d62f85f6957a-kube-api-access-scgrz\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:51.041827 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.041801    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-xks-ca" pod "c80d83c8-6b81-458b-9507-ed1f6f0c4d63" (UID: "c80d83c8-6b81-458b-9507-ed1f6f0c4d63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:51.041937 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.041890    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-kube-api-access-9r6vf" pod "c80d83c8-6b81-458b-9507-ed1f6f0c4d63" (UID: "c80d83c8-6b81-458b-9507-ed1f6f0c4d63"). InnerVolumeSpecName "kube-api-access-9r6vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:51.140878 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.140833    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:51.141093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.140878    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:51.141093 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.140912    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r6vf\" (UniqueName: \"kubernetes.io/projected/c80d83c8-6b81-458b-9507-ed1f6f0c4d63-kube-api-access-9r6vf\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:51.721807 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.721760    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck" event={"ID":"c80d83c8-6b81-458b-9507-ed1f6f0c4d63","Type":"ContainerDied","Data":"4e4814ede41c8297e19731db5d9bfc66aaf0c32ca0a0f841601f8409f0c8715d"}
Nov 01 12:57:51.722262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.721848    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e4814ede41c8297e19731db5d9bfc66aaf0c32ca0a0f841601f8409f0c8715d"
Nov 01 12:57:51.722262 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.721767    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"
Nov 01 12:57:51.723240 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.723215    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"
Nov 01 12:57:51.723383 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.723237    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs" event={"ID":"58425220-74da-46a5-ae9b-d62f85f6957a","Type":"ContainerDied","Data":"a90562ca58173849b21d6e13196afa58547546c5ca0ba48ca910e35c184a7288"}
Nov 01 12:57:51.723383 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.723274    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a90562ca58173849b21d6e13196afa58547546c5ca0ba48ca910e35c184a7288"
Nov 01 12:57:51.855362 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.855333    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:51.951974 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.951942    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-kube-api-access-hg2l8\" (UniqueName: \"kubernetes.io/projected/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-kube-api-access-hg2l8\") pod \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") "
Nov 01 12:57:51.952159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.952009    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-xks-ca\") pod \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") "
Nov 01 12:57:51.952159 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.952154    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-openshift-ca\") pod \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\" (UID: \"88057d9b-59f2-4e5f-8f1c-8057398cd2f9\") "
Nov 01 12:57:51.952553 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.952520    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-openshift-ca" pod "88057d9b-59f2-4e5f-8f1c-8057398cd2f9" (UID: "88057d9b-59f2-4e5f-8f1c-8057398cd2f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:51.954109 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.954074    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-kube-api-access-hg2l8" pod "88057d9b-59f2-4e5f-8f1c-8057398cd2f9" (UID: "88057d9b-59f2-4e5f-8f1c-8057398cd2f9"). InnerVolumeSpecName "kube-api-access-hg2l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:51.954210 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:51.954106    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-xks-ca" pod "88057d9b-59f2-4e5f-8f1c-8057398cd2f9" (UID: "88057d9b-59f2-4e5f-8f1c-8057398cd2f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:52.054265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:52.054193    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2l8\" (UniqueName: \"kubernetes.io/projected/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-kube-api-access-hg2l8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:52.054265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:52.054224    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:52.054265 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:52.054236    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88057d9b-59f2-4e5f-8f1c-8057398cd2f9-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:52.735247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:52.735210    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"
Nov 01 12:57:52.735247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:52.735240    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8" event={"ID":"88057d9b-59f2-4e5f-8f1c-8057398cd2f9","Type":"ContainerDied","Data":"637b7e0ed1268da6e591faecd3571aad0d586851d5d8f6fa7e8a773116f2ff09"}
Nov 01 12:57:52.735777 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:57:52.735318    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="637b7e0ed1268da6e591faecd3571aad0d586851d5d8f6fa7e8a773116f2ff09"
Nov 01 12:58:01.067082 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:58:01.065182    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:58:01.767039 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:01.766998    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"]
Nov 01 12:58:01.874326 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:01.874235    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"]
Nov 01 12:58:02.273699 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.273664    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"]
Nov 01 12:58:02.274283 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.274264    5725 cpu_manager.go:414] "containerMap: removing container" podUID="58425220-74da-46a5-ae9b-d62f85f6957a" containerName="revoke-keys"
Nov 01 12:58:02.274283 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.274281    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58425220-74da-46a5-ae9b-d62f85f6957a" containerName="revoke-keys"
Nov 01 12:58:02.274453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.274304    5725 cpu_manager.go:414] "containerMap: removing container" podUID="88057d9b-59f2-4e5f-8f1c-8057398cd2f9" containerName="revoke-keys"
Nov 01 12:58:02.274453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.274309    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88057d9b-59f2-4e5f-8f1c-8057398cd2f9" containerName="revoke-keys"
Nov 01 12:58:02.274453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.274332    5725 cpu_manager.go:414] "containerMap: removing container" podUID="c80d83c8-6b81-458b-9507-ed1f6f0c4d63" containerName="revoke-keys"
Nov 01 12:58:02.274453 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.274339    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80d83c8-6b81-458b-9507-ed1f6f0c4d63" containerName="revoke-keys"
Nov 01 12:58:02.274771 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.274751    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="58425220-74da-46a5-ae9b-d62f85f6957a" containerName="revoke-keys"
Nov 01 12:58:02.274861 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.274775    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="88057d9b-59f2-4e5f-8f1c-8057398cd2f9" containerName="revoke-keys"
Nov 01 12:58:02.274861 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.274785    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80d83c8-6b81-458b-9507-ed1f6f0c4d63" containerName="revoke-keys"
Nov 01 12:58:02.278726 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.278705    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.328934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.328521    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 12:58:02.328934 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.328745    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 12:58:02.332950 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.332916    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"]
Nov 01 12:58:02.386703 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.386665    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh9c5\" (UniqueName: \"kubernetes.io/projected/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-kube-api-access-wh9c5\") pod \"maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.386931 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.386751    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.386931 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.386861    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.487946 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.487907    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.488152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.488040    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh9c5\" (UniqueName: \"kubernetes.io/projected/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-kube-api-access-wh9c5\") pod \"maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.488152 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.488127    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.488661 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.488641    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.490482 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.490464    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.543170 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.543079    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh9c5\" (UniqueName: \"kubernetes.io/projected/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-kube-api-access-wh9c5\") pod \"maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.593518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.593459    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:02.982487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:02.982452    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"]
Nov 01 12:58:02.988917 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:58:02.988883    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff7b9fe_1f97_42c8_8ee1_c6f79edc856c.slice/crio-6ce6bb1ffcbed98f1cffaedbc86d6f6ddbe0ddac665d7bb7cd40bc953e1cc290 WatchSource:0}: Error finding container 6ce6bb1ffcbed98f1cffaedbc86d6f6ddbe0ddac665d7bb7cd40bc953e1cc290: Status 404 returned error can't find the container with id 6ce6bb1ffcbed98f1cffaedbc86d6f6ddbe0ddac665d7bb7cd40bc953e1cc290
Nov 01 12:58:03.821828 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:03.821776    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds" event={"ID":"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c","Type":"ContainerStarted","Data":"e0a06ce363abcbf19a6e162846a5e766a6b663a9be8c59a4536030f38363d41d"}
Nov 01 12:58:03.822327 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:03.821845    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds" event={"ID":"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c","Type":"ContainerStarted","Data":"6ce6bb1ffcbed98f1cffaedbc86d6f6ddbe0ddac665d7bb7cd40bc953e1cc290"}
Nov 01 12:58:03.930604 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:03.930541    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds" podStartSLOduration=1.9305278879999999 podStartE2EDuration="1.930527888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:58:03.929419057 +0000 UTC m=+2125.436610275" watchObservedRunningTime="2026-11-01 12:58:03.930527888 +0000 UTC m=+2125.437719104"
Nov 01 12:58:04.831329 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:04.831290    5725 generic.go:356] "Generic (PLEG): container finished" podID="cff7b9fe-1f97-42c8-8ee1-c6f79edc856c" containerID="e0a06ce363abcbf19a6e162846a5e766a6b663a9be8c59a4536030f38363d41d" exitCode=0
Nov 01 12:58:04.831815 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:04.831368    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds" event={"ID":"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c","Type":"ContainerDied","Data":"e0a06ce363abcbf19a6e162846a5e766a6b663a9be8c59a4536030f38363d41d"}
Nov 01 12:58:06.008852 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.008820    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:06.134372 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.134287    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-kube-api-access-wh9c5\" (UniqueName: \"kubernetes.io/projected/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-kube-api-access-wh9c5\") pod \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") "
Nov 01 12:58:06.134560 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.134459    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-xks-ca\") pod \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") "
Nov 01 12:58:06.134560 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.134520    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-openshift-ca\") pod \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\" (UID: \"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c\") "
Nov 01 12:58:06.135136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.135106    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-openshift-ca" pod "cff7b9fe-1f97-42c8-8ee1-c6f79edc856c" (UID: "cff7b9fe-1f97-42c8-8ee1-c6f79edc856c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:58:06.136763 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.136730    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-xks-ca" pod "cff7b9fe-1f97-42c8-8ee1-c6f79edc856c" (UID: "cff7b9fe-1f97-42c8-8ee1-c6f79edc856c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:58:06.136888 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.136862    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-kube-api-access-wh9c5" pod "cff7b9fe-1f97-42c8-8ee1-c6f79edc856c" (UID: "cff7b9fe-1f97-42c8-8ee1-c6f79edc856c"). InnerVolumeSpecName "kube-api-access-wh9c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:58:06.236555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.236523    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:06.236555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.236550    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh9c5\" (UniqueName: \"kubernetes.io/projected/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-kube-api-access-wh9c5\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:06.236555 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.236561    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:06.767857 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.767810    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9" podUID="3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689" containerName="payload-processing" containerID="cri-o://68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e" gracePeriod=25
Nov 01 12:58:06.850684 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.850652    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"
Nov 01 12:58:06.850879 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.850659    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds" event={"ID":"cff7b9fe-1f97-42c8-8ee1-c6f79edc856c","Type":"ContainerDied","Data":"6ce6bb1ffcbed98f1cffaedbc86d6f6ddbe0ddac665d7bb7cd40bc953e1cc290"}
Nov 01 12:58:06.850879 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.850782    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce6bb1ffcbed98f1cffaedbc86d6f6ddbe0ddac665d7bb7cd40bc953e1cc290"
Nov 01 12:58:06.874813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:06.874713    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr" podUID="6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85" containerName="payload-pre-processing" containerID="cri-o://cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62" gracePeriod=25
Nov 01 12:58:07.045866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.045835    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:58:07.149475 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.149441    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-plugins-config-volume\") pod \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\" (UID: \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\") "
Nov 01 12:58:07.149638 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.149519    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-kube-api-access-h29rw\") pod \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\" (UID: \"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689\") "
Nov 01 12:58:07.149871 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.149849    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-plugins-config-volume" pod "3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689" (UID: "3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:58:07.150180 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.150160    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:07.151833 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.151808    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-kube-api-access-h29rw" pod "3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689" (UID: "3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689"). InnerVolumeSpecName "kube-api-access-h29rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:58:07.175324 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.175298    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:58:07.251462 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.251428    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-plugins-config-volume\") pod \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\" (UID: \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\") "
Nov 01 12:58:07.251692 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.251668    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-kube-api-access-tr9xr\" (UniqueName: \"kubernetes.io/projected/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-kube-api-access-tr9xr\") pod \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\" (UID: \"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85\") "
Nov 01 12:58:07.252030 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.252006    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-plugins-config-volume" pod "6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85" (UID: "6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:58:07.252337 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.252316    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:07.252427 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.252343    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689-kube-api-access-h29rw\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:07.254009 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.253977    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-kube-api-access-tr9xr" pod "6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85" (UID: "6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85"). InnerVolumeSpecName "kube-api-access-tr9xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:58:07.354273 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.354232    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr9xr\" (UniqueName: \"kubernetes.io/projected/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85-kube-api-access-tr9xr\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:07.859621 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.859577    5725 generic.go:356] "Generic (PLEG): container finished" podID="3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689" containerID="68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e" exitCode=0
Nov 01 12:58:07.859840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.859622    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"
Nov 01 12:58:07.859840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.859631    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9" event={"ID":"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689","Type":"ContainerDied","Data":"68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e"}
Nov 01 12:58:07.859840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.859699    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9" event={"ID":"3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689","Type":"ContainerDied","Data":"9ba0bf5b723c32625393e3a6c03a4f7e97492b5f297938cf588374c60a1b2207"}
Nov 01 12:58:07.859840 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.859731    5725 scope.go:122] "RemoveContainer" containerID="68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e"
Nov 01 12:58:07.861758 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.861733    5725 generic.go:356] "Generic (PLEG): container finished" podID="6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85" containerID="cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62" exitCode=0
Nov 01 12:58:07.862014 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.861793    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"
Nov 01 12:58:07.862177 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.861810    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr" event={"ID":"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85","Type":"ContainerDied","Data":"cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62"}
Nov 01 12:58:07.862177 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.862138    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr" event={"ID":"6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85","Type":"ContainerDied","Data":"515931271e0d8e986a3c2e946fdb08455e94e19747095068ffd18d6ba8318421"}
Nov 01 12:58:07.877097 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.876460    5725 scope.go:122] "RemoveContainer" containerID="68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e"
Nov 01 12:58:07.878847 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:58:07.878812    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e\": container with ID starting with 68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e not found: ID does not exist" containerID="68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e"
Nov 01 12:58:07.878982 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.878856    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e"} err="failed to get container status \"68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e\": rpc error: code = NotFound desc = could not find container \"68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e\": container with ID starting with 68ccc471a2a9327c63fdbeea4f50fd70615c5fde1160109ca895927d14b22b2e not found: ID does not exist"
Nov 01 12:58:07.878982 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.878884    5725 scope.go:122] "RemoveContainer" containerID="cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62"
Nov 01 12:58:07.888664 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.888642    5725 scope.go:122] "RemoveContainer" containerID="cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62"
Nov 01 12:58:07.888931 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:58:07.888914    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62\": container with ID starting with cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62 not found: ID does not exist" containerID="cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62"
Nov 01 12:58:07.888996 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:07.888937    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62"} err="failed to get container status \"cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62\": rpc error: code = NotFound desc = could not find container \"cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62\": container with ID starting with cc0749dbdcfe0fc5c06da64f391fd39dae448482c9be3436956bd68e66446f62 not found: ID does not exist"
Nov 01 12:58:08.016136 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:08.016102    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"]
Nov 01 12:58:08.064761 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:08.064726    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6124f9-866b7cbc6f-2zrr9"]
Nov 01 12:58:08.121735 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:08.121656    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"]
Nov 01 12:58:08.127920 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:08.127883    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-6124f9-59fcdf947f-fqpcr"]
Nov 01 12:58:08.619010 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:08.618972    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689" path="/var/lib/kubelet/pods/3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689/volumes"
Nov 01 12:58:08.619487 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:08.619470    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85" path="/var/lib/kubelet/pods/6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85/volumes"
Nov 01 12:58:14.695818 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:14.695772    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"]
Nov 01 12:58:14.806157 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:14.805573    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"]
Nov 01 12:58:19.696766 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:19.696724    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b" podUID="ca93cb82-56a8-4a3f-806d-674a21c9d346" containerName="payload-processing" containerID="cri-o://8e0079937668b7b082b33f8949d9d77df036f84d50c5de7877d99f26473fecdc" gracePeriod=25
Nov 01 12:58:19.806376 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:19.806331    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg" podUID="57a2e157-c261-483a-b537-48292fbb16e5" containerName="payload-pre-processing" containerID="cri-o://57de5166a640d1d8ecfaaac34f3057b6358ff96f7d2979ebc63efb0bc7adf479" gracePeriod=25
Nov 01 12:58:19.957025 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:19.956980    5725 generic.go:356] "Generic (PLEG): container finished" podID="ca93cb82-56a8-4a3f-806d-674a21c9d346" containerID="8e0079937668b7b082b33f8949d9d77df036f84d50c5de7877d99f26473fecdc" exitCode=0
Nov 01 12:58:19.957308 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:19.957285    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b" event={"ID":"ca93cb82-56a8-4a3f-806d-674a21c9d346","Type":"ContainerDied","Data":"8e0079937668b7b082b33f8949d9d77df036f84d50c5de7877d99f26473fecdc"}
Nov 01 12:58:19.959993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:19.959388    5725 generic.go:356] "Generic (PLEG): container finished" podID="57a2e157-c261-483a-b537-48292fbb16e5" containerID="57de5166a640d1d8ecfaaac34f3057b6358ff96f7d2979ebc63efb0bc7adf479" exitCode=0
Nov 01 12:58:19.959993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:19.959447    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg" event={"ID":"57a2e157-c261-483a-b537-48292fbb16e5","Type":"ContainerDied","Data":"57de5166a640d1d8ecfaaac34f3057b6358ff96f7d2979ebc63efb0bc7adf479"}
Nov 01 12:58:19.986286 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:19.985727    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:58:20.016088 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.016043    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca93cb82-56a8-4a3f-806d-674a21c9d346-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca93cb82-56a8-4a3f-806d-674a21c9d346-plugins-config-volume\") pod \"ca93cb82-56a8-4a3f-806d-674a21c9d346\" (UID: \"ca93cb82-56a8-4a3f-806d-674a21c9d346\") "
Nov 01 12:58:20.016276 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.016260    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca93cb82-56a8-4a3f-806d-674a21c9d346-kube-api-access-c5f9l\" (UniqueName: \"kubernetes.io/projected/ca93cb82-56a8-4a3f-806d-674a21c9d346-kube-api-access-c5f9l\") pod \"ca93cb82-56a8-4a3f-806d-674a21c9d346\" (UID: \"ca93cb82-56a8-4a3f-806d-674a21c9d346\") "
Nov 01 12:58:20.016814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.016446    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca93cb82-56a8-4a3f-806d-674a21c9d346-plugins-config-volume" pod "ca93cb82-56a8-4a3f-806d-674a21c9d346" (UID: "ca93cb82-56a8-4a3f-806d-674a21c9d346"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:58:20.016814 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.016724    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca93cb82-56a8-4a3f-806d-674a21c9d346-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:20.019201 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.019168    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca93cb82-56a8-4a3f-806d-674a21c9d346-kube-api-access-c5f9l" pod "ca93cb82-56a8-4a3f-806d-674a21c9d346" (UID: "ca93cb82-56a8-4a3f-806d-674a21c9d346"). InnerVolumeSpecName "kube-api-access-c5f9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:58:20.089630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.089604    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:58:20.117365 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.117333    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a2e157-c261-483a-b537-48292fbb16e5-kube-api-access-k7dmw\" (UniqueName: \"kubernetes.io/projected/57a2e157-c261-483a-b537-48292fbb16e5-kube-api-access-k7dmw\") pod \"57a2e157-c261-483a-b537-48292fbb16e5\" (UID: \"57a2e157-c261-483a-b537-48292fbb16e5\") "
Nov 01 12:58:20.117630 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.117494    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a2e157-c261-483a-b537-48292fbb16e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2e157-c261-483a-b537-48292fbb16e5-plugins-config-volume\") pod \"57a2e157-c261-483a-b537-48292fbb16e5\" (UID: \"57a2e157-c261-483a-b537-48292fbb16e5\") "
Nov 01 12:58:20.117994 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.117966    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5f9l\" (UniqueName: \"kubernetes.io/projected/ca93cb82-56a8-4a3f-806d-674a21c9d346-kube-api-access-c5f9l\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:20.118140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.118039    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a2e157-c261-483a-b537-48292fbb16e5-plugins-config-volume" pod "57a2e157-c261-483a-b537-48292fbb16e5" (UID: "57a2e157-c261-483a-b537-48292fbb16e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:58:20.119581 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.119552    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a2e157-c261-483a-b537-48292fbb16e5-kube-api-access-k7dmw" pod "57a2e157-c261-483a-b537-48292fbb16e5" (UID: "57a2e157-c261-483a-b537-48292fbb16e5"). InnerVolumeSpecName "kube-api-access-k7dmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:58:20.218913 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.218836    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57a2e157-c261-483a-b537-48292fbb16e5-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:20.218913 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.218866    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7dmw\" (UniqueName: \"kubernetes.io/projected/57a2e157-c261-483a-b537-48292fbb16e5-kube-api-access-k7dmw\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:20.970281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.970256    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"
Nov 01 12:58:20.970704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.970301    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg" event={"ID":"57a2e157-c261-483a-b537-48292fbb16e5","Type":"ContainerDied","Data":"64de82a1c9058a4de7ddaec401e15ef29d85131b8f49d1b77adaec11966ee77e"}
Nov 01 12:58:20.970704 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.970370    5725 scope.go:122] "RemoveContainer" containerID="57de5166a640d1d8ecfaaac34f3057b6358ff96f7d2979ebc63efb0bc7adf479"
Nov 01 12:58:20.972357 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.972331    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"
Nov 01 12:58:20.972584 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.972559    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b" event={"ID":"ca93cb82-56a8-4a3f-806d-674a21c9d346","Type":"ContainerDied","Data":"9cd55cedbb9cea6e0ff04aced0e0457d5a4fddc48eed5f471e871c30618bd0f5"}
Nov 01 12:58:20.981705 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:20.981692    5725 scope.go:122] "RemoveContainer" containerID="8e0079937668b7b082b33f8949d9d77df036f84d50c5de7877d99f26473fecdc"
Nov 01 12:58:21.125428 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:21.125399    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"]
Nov 01 12:58:21.172570 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:21.172536    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-94c13a-649d679477-l9j2b"]
Nov 01 12:58:21.234126 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:21.232552    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"]
Nov 01 12:58:21.240813 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:21.240776    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-94c13a-74978db6bc-p8gjg"]
Nov 01 12:58:22.617404 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:22.617370    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a2e157-c261-483a-b537-48292fbb16e5" path="/var/lib/kubelet/pods/57a2e157-c261-483a-b537-48292fbb16e5/volumes"
Nov 01 12:58:22.617789 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:22.617780    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca93cb82-56a8-4a3f-806d-674a21c9d346" path="/var/lib/kubelet/pods/ca93cb82-56a8-4a3f-806d-674a21c9d346/volumes"
Nov 01 12:58:25.901008 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:25.900972    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"]
Nov 01 12:58:25.948223 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:25.948191    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6c650b-q2lck"]
Nov 01 12:58:26.619119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:26.619078    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80d83c8-6b81-458b-9507-ed1f6f0c4d63" path="/var/lib/kubelet/pods/c80d83c8-6b81-458b-9507-ed1f6f0c4d63/volumes"
Nov 01 12:58:31.031450 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:58:31.031415    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:58:32.925470 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:32.925436    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"]
Nov 01 12:58:32.925993 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:32.925649    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy" containerID="cri-o://67e7401149b4f83af7d991d52908b9f27fb8a2b22296e39da4924380edcf423a" gracePeriod=30
Nov 01 12:58:35.809226 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:35.809192    5725 patch_prober.go:29] interesting pod/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:58:35.809636 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:35.809259    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:58:38.144876 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.144833    5725 generic.go:356] "Generic (PLEG): container finished" podID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerID="67e7401149b4f83af7d991d52908b9f27fb8a2b22296e39da4924380edcf423a" exitCode=0
Nov 01 12:58:38.145367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.144901    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" event={"ID":"800fc942-dd8e-41df-8754-55ae9ab9cf46","Type":"ContainerDied","Data":"67e7401149b4f83af7d991d52908b9f27fb8a2b22296e39da4924380edcf423a"}
Nov 01 12:58:38.145367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.144981    5725 scope.go:122] "RemoveContainer" containerID="04758c8923fdf57090c3dec06241d0b93ccd419606908b7f3be00ff8a16f4406"
Nov 01 12:58:38.198672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.198603    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:58:38.350345 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350313    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-token\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-token\") pod \"800fc942-dd8e-41df-8754-55ae9ab9cf46\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") "
Nov 01 12:58:38.350518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350370    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-envoy\") pod \"800fc942-dd8e-41df-8754-55ae9ab9cf46\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") "
Nov 01 12:58:38.350518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350397    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-podinfo\") pod \"800fc942-dd8e-41df-8754-55ae9ab9cf46\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") "
Nov 01 12:58:38.350518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350437    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-kube-api-access-hxlhw\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-kube-api-access-hxlhw\") pod \"800fc942-dd8e-41df-8754-55ae9ab9cf46\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") "
Nov 01 12:58:38.350518 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350481    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800fc942-dd8e-41df-8754-55ae9ab9cf46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fc942-dd8e-41df-8754-55ae9ab9cf46-istiod-ca-cert\") pod \"800fc942-dd8e-41df-8754-55ae9ab9cf46\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") "
Nov 01 12:58:38.350767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350526    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-certs\") pod \"800fc942-dd8e-41df-8754-55ae9ab9cf46\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") "
Nov 01 12:58:38.350767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350605    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-data\") pod \"800fc942-dd8e-41df-8754-55ae9ab9cf46\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") "
Nov 01 12:58:38.350767 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350744    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-credential-socket\") pod \"800fc942-dd8e-41df-8754-55ae9ab9cf46\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") "
Nov 01 12:58:38.351007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350834    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-socket\") pod \"800fc942-dd8e-41df-8754-55ae9ab9cf46\" (UID: \"800fc942-dd8e-41df-8754-55ae9ab9cf46\") "
Nov 01 12:58:38.351007 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350953    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-certs" pod "800fc942-dd8e-41df-8754-55ae9ab9cf46" (UID: "800fc942-dd8e-41df-8754-55ae9ab9cf46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:58:38.351155 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.350997    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800fc942-dd8e-41df-8754-55ae9ab9cf46-istiod-ca-cert" pod "800fc942-dd8e-41df-8754-55ae9ab9cf46" (UID: "800fc942-dd8e-41df-8754-55ae9ab9cf46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:58:38.351437 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.351410    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/800fc942-dd8e-41df-8754-55ae9ab9cf46-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:38.351528 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.351442    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:38.351736 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.351710    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-credential-socket" pod "800fc942-dd8e-41df-8754-55ae9ab9cf46" (UID: "800fc942-dd8e-41df-8754-55ae9ab9cf46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:58:38.351825 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.351761    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-socket" pod "800fc942-dd8e-41df-8754-55ae9ab9cf46" (UID: "800fc942-dd8e-41df-8754-55ae9ab9cf46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:58:38.351825 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.351791    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-data" pod "800fc942-dd8e-41df-8754-55ae9ab9cf46" (UID: "800fc942-dd8e-41df-8754-55ae9ab9cf46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:58:38.352698 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.352672    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-podinfo" pod "800fc942-dd8e-41df-8754-55ae9ab9cf46" (UID: "800fc942-dd8e-41df-8754-55ae9ab9cf46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:58:38.352770 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.352727    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-token" pod "800fc942-dd8e-41df-8754-55ae9ab9cf46" (UID: "800fc942-dd8e-41df-8754-55ae9ab9cf46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:58:38.352770 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.352744    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-envoy" pod "800fc942-dd8e-41df-8754-55ae9ab9cf46" (UID: "800fc942-dd8e-41df-8754-55ae9ab9cf46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:58:38.352869 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.352853    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-kube-api-access-hxlhw" pod "800fc942-dd8e-41df-8754-55ae9ab9cf46" (UID: "800fc942-dd8e-41df-8754-55ae9ab9cf46"). InnerVolumeSpecName "kube-api-access-hxlhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:58:38.453169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.453075    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:38.453169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.453104    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:38.453169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.453114    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:38.453169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.453123    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:38.453169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.453132    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:38.453169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.453141    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxlhw\" (UniqueName: \"kubernetes.io/projected/800fc942-dd8e-41df-8754-55ae9ab9cf46-kube-api-access-hxlhw\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:38.453169 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:38.453150    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/800fc942-dd8e-41df-8754-55ae9ab9cf46-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:39.154515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:39.154488    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"
Nov 01 12:58:39.154952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:39.154538    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88" event={"ID":"800fc942-dd8e-41df-8754-55ae9ab9cf46","Type":"ContainerDied","Data":"b9443dc3b35550270e955b1f997917f31d4f1fbf1b8011df114d261f95f5bfbb"}
Nov 01 12:58:39.154952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:39.154596    5725 scope.go:122] "RemoveContainer" containerID="67e7401149b4f83af7d991d52908b9f27fb8a2b22296e39da4924380edcf423a"
Nov 01 12:58:39.358298 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:39.358263    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"]
Nov 01 12:58:39.405269 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:39.405180    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-6c650b-openshift-default-bbdf5f89c-ljj88"]
Nov 01 12:58:40.616945 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:40.616906    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" path="/var/lib/kubelet/pods/800fc942-dd8e-41df-8754-55ae9ab9cf46/volumes"
Nov 01 12:58:48.321419 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.321383    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"]
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.321935    5725 cpu_manager.go:414] "containerMap: removing container" podUID="ca93cb82-56a8-4a3f-806d-674a21c9d346" containerName="payload-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.321947    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca93cb82-56a8-4a3f-806d-674a21c9d346" containerName="payload-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.321960    5725 cpu_manager.go:414] "containerMap: removing container" podUID="cff7b9fe-1f97-42c8-8ee1-c6f79edc856c" containerName="revoke-keys"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.321965    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff7b9fe-1f97-42c8-8ee1-c6f79edc856c" containerName="revoke-keys"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.321975    5725 cpu_manager.go:414] "containerMap: removing container" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.321981    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.321994    5725 cpu_manager.go:414] "containerMap: removing container" podUID="3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689" containerName="payload-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.321998    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689" containerName="payload-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322018    5725 cpu_manager.go:414] "containerMap: removing container" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322022    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322031    5725 cpu_manager.go:414] "containerMap: removing container" podUID="6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85" containerName="payload-pre-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322036    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85" containerName="payload-pre-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322072    5725 cpu_manager.go:414] "containerMap: removing container" podUID="57a2e157-c261-483a-b537-48292fbb16e5" containerName="payload-pre-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322079    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a2e157-c261-483a-b537-48292fbb16e5" containerName="payload-pre-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322162    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322170    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="6685a3f8-4bf1-485f-a6bc-8e9eb0f1cd85" containerName="payload-pre-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322178    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a2e157-c261-483a-b537-48292fbb16e5" containerName="payload-pre-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322186    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca93cb82-56a8-4a3f-806d-674a21c9d346" containerName="payload-processing"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322191    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff7b9fe-1f97-42c8-8ee1-c6f79edc856c" containerName="revoke-keys"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322197    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="800fc942-dd8e-41df-8754-55ae9ab9cf46" containerName="istio-proxy"
Nov 01 12:58:48.323849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.322204    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8d1a41-d6cc-4243-ad9b-6b3c6efa2689" containerName="payload-processing"
Nov 01 12:58:48.325646 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.325630    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.378441 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.378410    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.378600 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.378466    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5jcd\" (UniqueName: \"kubernetes.io/projected/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-kube-api-access-v5jcd\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.378685 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.378657    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-metrics-certs\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.378774 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.378743    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.422707 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.422678    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-72078a-serving-cert\""
Nov 01 12:58:48.426284 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.426248    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"]
Nov 01 12:58:48.479534 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.479501    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-metrics-certs\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.479715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.479545    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.479715 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.479671    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.479849 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.479729    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5jcd\" (UniqueName: \"kubernetes.io/projected/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-kube-api-access-v5jcd\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.479849 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:58:48.479820    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-serving-cert: secret "maas-api-e2e-inf-a-72078a-serving-cert" not found
Nov 01 12:58:48.479969 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:58:48.479919    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls podName:b0ad545d-8951-47b1-887d-8c7f9bfd43cc nodeName:}" failed. No retries permitted until 2026-11-01 12:58:48.97990009 +0000 UTC m=+2170.487091304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls") pod "maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt" (UID: "b0ad545d-8951-47b1-887d-8c7f9bfd43cc") : secret "maas-api-e2e-inf-a-72078a-serving-cert" not found
Nov 01 12:58:48.480377 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.480354    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.483125 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.483102    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-metrics-certs\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.536149 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.534890    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5jcd\" (UniqueName: \"kubernetes.io/projected/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-kube-api-access-v5jcd\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.985087 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.984979    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:48.987431 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:48.987407    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls\") pod \"maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:49.242293 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:49.242205    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:49.623965 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:49.623887    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"]
Nov 01 12:58:49.631422 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:58:49.631392    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ad545d_8951_47b1_887d_8c7f9bfd43cc.slice/crio-f23d0f786d84d673287247cfb56a567d96a9d820c96e337030ff6d30ce5a1c26 WatchSource:0}: Error finding container f23d0f786d84d673287247cfb56a567d96a9d820c96e337030ff6d30ce5a1c26: Status 404 returned error can't find the container with id f23d0f786d84d673287247cfb56a567d96a9d820c96e337030ff6d30ce5a1c26
Nov 01 12:58:50.239759 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:50.239717    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt" event={"ID":"b0ad545d-8951-47b1-887d-8c7f9bfd43cc","Type":"ContainerStarted","Data":"f23d0f786d84d673287247cfb56a567d96a9d820c96e337030ff6d30ce5a1c26"}
Nov 01 12:58:51.250232 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:51.250190    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt" event={"ID":"b0ad545d-8951-47b1-887d-8c7f9bfd43cc","Type":"ContainerStarted","Data":"4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5"}
Nov 01 12:58:51.250611 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:51.250268    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:58:51.360759 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:51.360707    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt" podStartSLOduration=2.378036749 podStartE2EDuration="3.360693285s" totalImagesPullingTime="982.656536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:58:51.358096249 +0000 UTC m=+2172.865287470" watchObservedRunningTime="2026-11-01 12:58:51.360693285 +0000 UTC m=+2172.867884523"
Nov 01 12:58:57.263511 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:58:57.263471    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 12:59:01.028537 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:01.028500    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:59:01.170432 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.170363    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"]
Nov 01 12:59:01.170743 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.170628    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q" podUID="1a8dbd25-9551-4b82-931e-dfda55ba2214" containerName="maas-api" containerID="cri-o://46b7cc0afa25149826245cff728bfd31923e19f34c542fc3afb1b13c56551247" gracePeriod=30
Nov 01 12:59:01.335896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.335576    5725 generic.go:356] "Generic (PLEG): container finished" podID="1a8dbd25-9551-4b82-931e-dfda55ba2214" containerID="46b7cc0afa25149826245cff728bfd31923e19f34c542fc3afb1b13c56551247" exitCode=0
Nov 01 12:59:01.335896 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.335761    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q" event={"ID":"1a8dbd25-9551-4b82-931e-dfda55ba2214","Type":"ContainerDied","Data":"46b7cc0afa25149826245cff728bfd31923e19f34c542fc3afb1b13c56551247"}
Nov 01 12:59:01.516076 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.516029    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:59:01.668077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.666490    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8dbd25-9551-4b82-931e-dfda55ba2214-kube-api-access-ssdxn\" (UniqueName: \"kubernetes.io/projected/1a8dbd25-9551-4b82-931e-dfda55ba2214-kube-api-access-ssdxn\") pod \"1a8dbd25-9551-4b82-931e-dfda55ba2214\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") "
Nov 01 12:59:01.668077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.666578    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-metrics-certs\") pod \"1a8dbd25-9551-4b82-931e-dfda55ba2214\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") "
Nov 01 12:59:01.668077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.666743    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8dbd25-9551-4b82-931e-dfda55ba2214-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dbd25-9551-4b82-931e-dfda55ba2214-openshift-service-ca\") pod \"1a8dbd25-9551-4b82-931e-dfda55ba2214\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") "
Nov 01 12:59:01.668077 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.666773    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls\") pod \"1a8dbd25-9551-4b82-931e-dfda55ba2214\" (UID: \"1a8dbd25-9551-4b82-931e-dfda55ba2214\") "
Nov 01 12:59:01.687137 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.680244    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8dbd25-9551-4b82-931e-dfda55ba2214-openshift-service-ca" pod "1a8dbd25-9551-4b82-931e-dfda55ba2214" (UID: "1a8dbd25-9551-4b82-931e-dfda55ba2214"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:59:01.703658 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.703610    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-metrics-certs" pod "1a8dbd25-9551-4b82-931e-dfda55ba2214" (UID: "1a8dbd25-9551-4b82-931e-dfda55ba2214"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:59:01.715281 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.715193    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls" pod "1a8dbd25-9551-4b82-931e-dfda55ba2214" (UID: "1a8dbd25-9551-4b82-931e-dfda55ba2214"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:59:01.717251 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.717219    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8dbd25-9551-4b82-931e-dfda55ba2214-kube-api-access-ssdxn" pod "1a8dbd25-9551-4b82-931e-dfda55ba2214" (UID: "1a8dbd25-9551-4b82-931e-dfda55ba2214"). InnerVolumeSpecName "kube-api-access-ssdxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:01.768893 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.768812    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:01.769242 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.769214    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8dbd25-9551-4b82-931e-dfda55ba2214-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:01.769390 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.769361    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a8dbd25-9551-4b82-931e-dfda55ba2214-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:01.769601 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:01.769554    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssdxn\" (UniqueName: \"kubernetes.io/projected/1a8dbd25-9551-4b82-931e-dfda55ba2214-kube-api-access-ssdxn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:02.351684 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:02.351646    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q" event={"ID":"1a8dbd25-9551-4b82-931e-dfda55ba2214","Type":"ContainerDied","Data":"6f1dd31715cfd90cb890681fd4093584ec93156ff584e5d7d635113b96c90b8a"}
Nov 01 12:59:02.352190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:02.351709    5725 scope.go:122] "RemoveContainer" containerID="46b7cc0afa25149826245cff728bfd31923e19f34c542fc3afb1b13c56551247"
Nov 01 12:59:02.352190 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:02.351720    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"
Nov 01 12:59:02.559904 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:02.559869    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"]
Nov 01 12:59:02.607423 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:02.607334    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-6124f9-5cb67994f8-sxz4q"]
Nov 01 12:59:02.618844 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:02.618807    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8dbd25-9551-4b82-931e-dfda55ba2214" path="/var/lib/kubelet/pods/1a8dbd25-9551-4b82-931e-dfda55ba2214/volumes"
Nov 01 12:59:03.700983 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:03.700946    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"]
Nov 01 12:59:03.735596 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:03.735558    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6124f9-g4bvs"]
Nov 01 12:59:04.617902 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:04.617863    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58425220-74da-46a5-ae9b-d62f85f6957a" path="/var/lib/kubelet/pods/58425220-74da-46a5-ae9b-d62f85f6957a/volumes"
Nov 01 12:59:10.434760 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:10.434724    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"]
Nov 01 12:59:10.435331 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:10.435003    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy" containerID="cri-o://20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26" gracePeriod=30
Nov 01 12:59:12.127271 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:12.127233    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"]
Nov 01 12:59:12.127775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:12.127479    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy" containerID="cri-o://1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a" gracePeriod=30
Nov 01 12:59:15.706226 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.706161    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:59:15.753929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.753893    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-token\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-token\") pod \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") "
Nov 01 12:59:15.754140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754002    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-kube-api-access-kltd8\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-kube-api-access-kltd8\") pod \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") "
Nov 01 12:59:15.754140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754090    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istiod-ca-cert\") pod \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") "
Nov 01 12:59:15.754140 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754127    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-envoy\") pod \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") "
Nov 01 12:59:15.754343 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754261    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-certs\") pod \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") "
Nov 01 12:59:15.754425 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754403    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-socket\") pod \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") "
Nov 01 12:59:15.754485 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754454    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-podinfo\") pod \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") "
Nov 01 12:59:15.754538 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754523    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-data\") pod \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") "
Nov 01 12:59:15.754598 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754577    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-credential-socket\") pod \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\" (UID: \"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9\") "
Nov 01 12:59:15.754892 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754733    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istiod-ca-cert" pod "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" (UID: "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:59:15.755006 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.754935    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-credential-socket" pod "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" (UID: "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:15.755373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.755191    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-socket" pod "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" (UID: "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:15.755373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.755226    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-data" pod "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" (UID: "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:15.755373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.755245    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-certs" pod "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" (UID: "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:15.755373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.755271    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:15.755373 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.755297    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:15.757500 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.757476    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-token" pod "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" (UID: "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:15.757602 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.757510    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-kube-api-access-kltd8" pod "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" (UID: "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9"). InnerVolumeSpecName "kube-api-access-kltd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:15.757676 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.757617    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-podinfo" pod "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" (UID: "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:59:15.757786 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.757763    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-envoy" pod "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" (UID: "da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:15.856033 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.856001    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kltd8\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-kube-api-access-kltd8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:15.856033 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.856029    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:15.856033 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.856041    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:15.856033 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.856063    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:15.856335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.856077    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:15.856335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.856088    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:15.856335 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:15.856097    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:16.471412 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.471375    5725 generic.go:356] "Generic (PLEG): container finished" podID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerID="20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26" exitCode=0
Nov 01 12:59:16.471605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.471421    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"
Nov 01 12:59:16.471605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.471462    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" event={"ID":"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9","Type":"ContainerDied","Data":"20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26"}
Nov 01 12:59:16.471605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.471524    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl" event={"ID":"da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9","Type":"ContainerDied","Data":"d8ad0979c19fe0a0787a1bb6a29e93d4dee12cb3dc6c7eef43eaa11ee490b52f"}
Nov 01 12:59:16.471605 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.471555    5725 scope.go:122] "RemoveContainer" containerID="20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26"
Nov 01 12:59:16.483746 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.483716    5725 scope.go:122] "RemoveContainer" containerID="e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a"
Nov 01 12:59:16.493641 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.493617    5725 scope.go:122] "RemoveContainer" containerID="20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26"
Nov 01 12:59:16.493931 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:16.493912    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26\": container with ID starting with 20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26 not found: ID does not exist" containerID="20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26"
Nov 01 12:59:16.493983 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.493941    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26"} err="failed to get container status \"20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26\": rpc error: code = NotFound desc = could not find container \"20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26\": container with ID starting with 20cf81d00d8ec63ab9152e0bbbbcc514185099423c63497e61f073e10172bc26 not found: ID does not exist"
Nov 01 12:59:16.493983 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.493958    5725 scope.go:122] "RemoveContainer" containerID="e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a"
Nov 01 12:59:16.494236 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:16.494217    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a\": container with ID starting with e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a not found: ID does not exist" containerID="e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a"
Nov 01 12:59:16.494300 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.494245    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a"} err="failed to get container status \"e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a\": rpc error: code = NotFound desc = could not find container \"e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a\": container with ID starting with e97f10b9435f5d7038c937805b46c7644d7a91d0595b06c58c980dcf7ca9d88a not found: ID does not exist"
Nov 01 12:59:16.629247 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.629215    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"]
Nov 01 12:59:16.675754 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:16.675716    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-6124f9-openshift-default-5b766695b7-w87pl"]
Nov 01 12:59:17.411172 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.411112    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:59:17.480642 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.480604    5725 generic.go:356] "Generic (PLEG): container finished" podID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerID="1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a" exitCode=0
Nov 01 12:59:17.480866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.480658    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"
Nov 01 12:59:17.480866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.480688    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" event={"ID":"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18","Type":"ContainerDied","Data":"1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a"}
Nov 01 12:59:17.480866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.480741    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h" event={"ID":"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18","Type":"ContainerDied","Data":"d7330fa6ef749d2f6d48ede2cc9b69104e50d4801b396186d16e54d5f7fb34d2"}
Nov 01 12:59:17.480866 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.480765    5725 scope.go:122] "RemoveContainer" containerID="1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a"
Nov 01 12:59:17.484119 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484096    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-certs\") pod \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") "
Nov 01 12:59:17.484275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484153    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-token\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-token\") pod \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") "
Nov 01 12:59:17.484275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484209    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istiod-ca-cert\") pod \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") "
Nov 01 12:59:17.484275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484257    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-kube-api-access-c4vr4\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-kube-api-access-c4vr4\") pod \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") "
Nov 01 12:59:17.484450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484297    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-envoy\") pod \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") "
Nov 01 12:59:17.484450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484422    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-data\") pod \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") "
Nov 01 12:59:17.484450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484430    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-certs" pod "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" (UID: "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.484597 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484464    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-socket\") pod \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") "
Nov 01 12:59:17.484597 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484587    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-podinfo\") pod \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") "
Nov 01 12:59:17.484703 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484616    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istiod-ca-cert" pod "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" (UID: "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:59:17.484762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484719    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-data" pod "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" (UID: "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.484762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484733    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-socket" pod "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" (UID: "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.484762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484620    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-credential-socket\") pod \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\" (UID: \"aeaaaf7c-b849-44e5-8ea3-d042a0bbed18\") "
Nov 01 12:59:17.484915 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.484870    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-credential-socket" pod "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" (UID: "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.485217 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.485197    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.485328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.485218    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.485328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.485230    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.485328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.485239    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.485328 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.485247    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.486828 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.486806    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-podinfo" pod "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" (UID: "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:59:17.486929 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.486805    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-kube-api-access-c4vr4" pod "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" (UID: "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18"). InnerVolumeSpecName "kube-api-access-c4vr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:17.486989 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.486970    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-envoy" pod "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" (UID: "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.488450 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.488432    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-token" pod "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" (UID: "aeaaaf7c-b849-44e5-8ea3-d042a0bbed18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:17.492794 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.492776    5725 scope.go:122] "RemoveContainer" containerID="1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a"
Nov 01 12:59:17.493134 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:17.493102    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a\": container with ID starting with 1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a not found: ID does not exist" containerID="1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a"
Nov 01 12:59:17.493248 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.493143    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a"} err="failed to get container status \"1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a\": rpc error: code = NotFound desc = could not find container \"1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a\": container with ID starting with 1263d5c6c1b4b16e4a66fc24fb5da1ba4d328ca6207b9bef743315698acf206a not found: ID does not exist"
Nov 01 12:59:17.585868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.585832    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.585868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.585859    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.585868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.585869    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vr4\" (UniqueName: \"kubernetes.io/projected/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-kube-api-access-c4vr4\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.585868 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.585879    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.901312 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.901258    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"]
Nov 01 12:59:17.946341 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:17.946309    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-6h76h"]
Nov 01 12:59:18.616848 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:18.616818    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" path="/var/lib/kubelet/pods/aeaaaf7c-b849-44e5-8ea3-d042a0bbed18/volumes"
Nov 01 12:59:18.617310 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:18.617295    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" path="/var/lib/kubelet/pods/da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9/volumes"
Nov 01 12:59:31.034368 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:31.034325    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 12:59:41.674092 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:41.674041    5725 scope.go:122] "RemoveContainer" containerID="697b958f1ba7d8abda7d44571353f126994902c73f06452016527518a96ae9d5"
Nov 01 12:59:43.273094 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.273033    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"]
Nov 01 12:59:43.273556 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.273294    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp" podUID="193c5d61-aea3-44c8-9d2b-0236899753b9" containerName="maas-api" containerID="cri-o://8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c" gracePeriod=30
Nov 01 12:59:43.577928 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.577899    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:59:43.717529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.711566    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls\") pod \"193c5d61-aea3-44c8-9d2b-0236899753b9\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") "
Nov 01 12:59:43.717529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.711629    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193c5d61-aea3-44c8-9d2b-0236899753b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193c5d61-aea3-44c8-9d2b-0236899753b9-openshift-service-ca\") pod \"193c5d61-aea3-44c8-9d2b-0236899753b9\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") "
Nov 01 12:59:43.717529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.711671    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-metrics-certs\") pod \"193c5d61-aea3-44c8-9d2b-0236899753b9\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") "
Nov 01 12:59:43.717529 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.711819    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193c5d61-aea3-44c8-9d2b-0236899753b9-kube-api-access-6nwzc\" (UniqueName: \"kubernetes.io/projected/193c5d61-aea3-44c8-9d2b-0236899753b9-kube-api-access-6nwzc\") pod \"193c5d61-aea3-44c8-9d2b-0236899753b9\" (UID: \"193c5d61-aea3-44c8-9d2b-0236899753b9\") "
Nov 01 12:59:43.720758 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.719626    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193c5d61-aea3-44c8-9d2b-0236899753b9-openshift-service-ca" pod "193c5d61-aea3-44c8-9d2b-0236899753b9" (UID: "193c5d61-aea3-44c8-9d2b-0236899753b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:59:43.736703 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.723579    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193c5d61-aea3-44c8-9d2b-0236899753b9-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:43.741410 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.741356    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193c5d61-aea3-44c8-9d2b-0236899753b9-kube-api-access-6nwzc" pod "193c5d61-aea3-44c8-9d2b-0236899753b9" (UID: "193c5d61-aea3-44c8-9d2b-0236899753b9"). InnerVolumeSpecName "kube-api-access-6nwzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:43.758416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.751211    5725 generic.go:356] "Generic (PLEG): container finished" podID="193c5d61-aea3-44c8-9d2b-0236899753b9" containerID="8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c" exitCode=0
Nov 01 12:59:43.758416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.751469    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp" event={"ID":"193c5d61-aea3-44c8-9d2b-0236899753b9","Type":"ContainerDied","Data":"8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c"}
Nov 01 12:59:43.758416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.751525    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp" event={"ID":"193c5d61-aea3-44c8-9d2b-0236899753b9","Type":"ContainerDied","Data":"108fe807de917f2db7618b853037517250653f40a12a9a9dee05329b561a772f"}
Nov 01 12:59:43.758416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.751556    5725 scope.go:122] "RemoveContainer" containerID="8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c"
Nov 01 12:59:43.758416 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.751733    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"
Nov 01 12:59:43.775307 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.775252    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls" pod "193c5d61-aea3-44c8-9d2b-0236899753b9" (UID: "193c5d61-aea3-44c8-9d2b-0236899753b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:59:43.776719 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.776685    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-metrics-certs" pod "193c5d61-aea3-44c8-9d2b-0236899753b9" (UID: "193c5d61-aea3-44c8-9d2b-0236899753b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:59:43.786367 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.786347    5725 scope.go:122] "RemoveContainer" containerID="8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c"
Nov 01 12:59:43.786706 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:43.786681    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c\": container with ID starting with 8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c not found: ID does not exist" containerID="8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c"
Nov 01 12:59:43.786788 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.786716    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c"} err="failed to get container status \"8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c\": rpc error: code = NotFound desc = could not find container \"8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c\": container with ID starting with 8be5c9538d601e70ff7a95105d84f17b7b1c61330b218336dfa00fab5a883a3c not found: ID does not exist"
Nov 01 12:59:43.824541 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.824437    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nwzc\" (UniqueName: \"kubernetes.io/projected/193c5d61-aea3-44c8-9d2b-0236899753b9-kube-api-access-6nwzc\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:43.824541 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.824479    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:43.824541 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:43.824494    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/193c5d61-aea3-44c8-9d2b-0236899753b9-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:44.173775 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:44.173691    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"]
Nov 01 12:59:44.178595 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:44.178564    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0cd09f-7b4454f478-w95tp"]
Nov 01 12:59:44.616289 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:44.616256    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193c5d61-aea3-44c8-9d2b-0236899753b9" path="/var/lib/kubelet/pods/193c5d61-aea3-44c8-9d2b-0236899753b9/volumes"
Nov 01 12:59:54.096721 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:54.096685    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"]
Nov 01 12:59:54.097318 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:54.096951    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy" containerID="cri-o://c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8" gracePeriod=30
Nov 01 12:59:56.910811 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.910775    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"]
Nov 01 12:59:56.911337 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911322    5725 cpu_manager.go:414] "containerMap: removing container" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy"
Nov 01 12:59:56.911389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911337    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy"
Nov 01 12:59:56.911389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911356    5725 cpu_manager.go:414] "containerMap: removing container" podUID="1a8dbd25-9551-4b82-931e-dfda55ba2214" containerName="maas-api"
Nov 01 12:59:56.911389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911361    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8dbd25-9551-4b82-931e-dfda55ba2214" containerName="maas-api"
Nov 01 12:59:56.911389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911375    5725 cpu_manager.go:414] "containerMap: removing container" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy"
Nov 01 12:59:56.911389 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911381    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy"
Nov 01 12:59:56.911548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911407    5725 cpu_manager.go:414] "containerMap: removing container" podUID="193c5d61-aea3-44c8-9d2b-0236899753b9" containerName="maas-api"
Nov 01 12:59:56.911548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911412    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193c5d61-aea3-44c8-9d2b-0236899753b9" containerName="maas-api"
Nov 01 12:59:56.911548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911489    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeaaaf7c-b849-44e5-8ea3-d042a0bbed18" containerName="istio-proxy"
Nov 01 12:59:56.911548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911498    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy"
Nov 01 12:59:56.911548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911508    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="193c5d61-aea3-44c8-9d2b-0236899753b9" containerName="maas-api"
Nov 01 12:59:56.911548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911515    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy"
Nov 01 12:59:56.911548 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.911522    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8dbd25-9551-4b82-931e-dfda55ba2214" containerName="maas-api"
Nov 01 12:59:56.916165 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.916135    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:56.970148 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:56.970112    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"]
Nov 01 12:59:57.001826 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.001789    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-metrics-certs\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.002023 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.001915    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2v5r\" (UniqueName: \"kubernetes.io/projected/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-kube-api-access-h2v5r\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.002023 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.001986    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.002187 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.002022    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-openshift-service-ca\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.012762 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.012716    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-187d06ed-serving-cert\""
Nov 01 12:59:57.103438 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.103399    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2v5r\" (UniqueName: \"kubernetes.io/projected/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-kube-api-access-h2v5r\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.103611 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.103491    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.103611 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.103524    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-openshift-service-ca\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.103611 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.103593    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-metrics-certs\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.103817 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:57.103636    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-serving-cert: secret "maas-api-e2e-mt-187d06ed-serving-cert" not found
Nov 01 12:59:57.103817 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:57.103717    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls podName:eaa6b7da-b631-4bd0-9487-073cf7ab1ec1 nodeName:}" failed. No retries permitted until 2026-11-01 12:59:57.603696673 +0000 UTC m=+2239.110887873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls") pod "maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9" (UID: "eaa6b7da-b631-4bd0-9487-073cf7ab1ec1") : secret "maas-api-e2e-mt-187d06ed-serving-cert" not found
Nov 01 12:59:57.104824 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.104791    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-openshift-service-ca\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.106567 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.106539    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-metrics-certs\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.165112 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.158916    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2v5r\" (UniqueName: \"kubernetes.io/projected/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-kube-api-access-h2v5r\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.609520 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.609479    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.611846 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.611817    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls\") pod \"maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:57.830229 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:57.830186    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:58.214473 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:58.214387    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"]
Nov 01 12:59:58.220040 ip-10-0-138-76 kubenswrapper[5725]: W1101 12:59:58.220012    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa6b7da_b631_4bd0_9487_073cf7ab1ec1.slice/crio-c992638911374a6549c4ff15f824408c7d3b7820f74ac948753736acfa5a9555 WatchSource:0}: Error finding container c992638911374a6549c4ff15f824408c7d3b7820f74ac948753736acfa5a9555: Status 404 returned error can't find the container with id c992638911374a6549c4ff15f824408c7d3b7820f74ac948753736acfa5a9555
Nov 01 12:59:58.877205 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:58.877165    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9" event={"ID":"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1","Type":"ContainerStarted","Data":"c992638911374a6549c4ff15f824408c7d3b7820f74ac948753736acfa5a9555"}
Nov 01 12:59:58.955132 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:58.955091    5725 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:59:58.955424 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:58.955388    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:59:59.375258 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.375232    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:59:59.432625 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.432521    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-certs\") pod \"c0b7d530-db61-4ec9-9647-deb92b22a82f\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") "
Nov 01 12:59:59.432625 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.432589    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-socket\") pod \"c0b7d530-db61-4ec9-9647-deb92b22a82f\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") "
Nov 01 12:59:59.432870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.432630    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b7d530-db61-4ec9-9647-deb92b22a82f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b7d530-db61-4ec9-9647-deb92b22a82f-istiod-ca-cert\") pod \"c0b7d530-db61-4ec9-9647-deb92b22a82f\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") "
Nov 01 12:59:59.432870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.432676    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-kube-api-access-gkzsx\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-kube-api-access-gkzsx\") pod \"c0b7d530-db61-4ec9-9647-deb92b22a82f\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") "
Nov 01 12:59:59.432870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.432771    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-data\") pod \"c0b7d530-db61-4ec9-9647-deb92b22a82f\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") "
Nov 01 12:59:59.432870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.432805    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-podinfo\") pod \"c0b7d530-db61-4ec9-9647-deb92b22a82f\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") "
Nov 01 12:59:59.432870 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.432823    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-certs" pod "c0b7d530-db61-4ec9-9647-deb92b22a82f" (UID: "c0b7d530-db61-4ec9-9647-deb92b22a82f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:59.433184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.432938    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-socket" pod "c0b7d530-db61-4ec9-9647-deb92b22a82f" (UID: "c0b7d530-db61-4ec9-9647-deb92b22a82f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:59.433184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433001    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-token\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-token\") pod \"c0b7d530-db61-4ec9-9647-deb92b22a82f\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") "
Nov 01 12:59:59.433184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433044    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-credential-socket\") pod \"c0b7d530-db61-4ec9-9647-deb92b22a82f\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") "
Nov 01 12:59:59.433184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433073    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b7d530-db61-4ec9-9647-deb92b22a82f-istiod-ca-cert" pod "c0b7d530-db61-4ec9-9647-deb92b22a82f" (UID: "c0b7d530-db61-4ec9-9647-deb92b22a82f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:59:59.433184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433107    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-envoy\") pod \"c0b7d530-db61-4ec9-9647-deb92b22a82f\" (UID: \"c0b7d530-db61-4ec9-9647-deb92b22a82f\") "
Nov 01 12:59:59.433184 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433167    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-data" pod "c0b7d530-db61-4ec9-9647-deb92b22a82f" (UID: "c0b7d530-db61-4ec9-9647-deb92b22a82f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:59.433515 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433364    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-credential-socket" pod "c0b7d530-db61-4ec9-9647-deb92b22a82f" (UID: "c0b7d530-db61-4ec9-9647-deb92b22a82f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:59.433675 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433655    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:59.433769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433680    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:59.433769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433695    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:59.433769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433707    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:59.433769 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.433724    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b7d530-db61-4ec9-9647-deb92b22a82f-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:59.435559 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.435529    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-token" pod "c0b7d530-db61-4ec9-9647-deb92b22a82f" (UID: "c0b7d530-db61-4ec9-9647-deb92b22a82f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:59.435817 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.435793    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-envoy" pod "c0b7d530-db61-4ec9-9647-deb92b22a82f" (UID: "c0b7d530-db61-4ec9-9647-deb92b22a82f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:59.435914 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.435825    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-kube-api-access-gkzsx" pod "c0b7d530-db61-4ec9-9647-deb92b22a82f" (UID: "c0b7d530-db61-4ec9-9647-deb92b22a82f"). InnerVolumeSpecName "kube-api-access-gkzsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:59.436040 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.436018    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-podinfo" pod "c0b7d530-db61-4ec9-9647-deb92b22a82f" (UID: "c0b7d530-db61-4ec9-9647-deb92b22a82f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:59:59.517682 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.517637    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"]
Nov 01 12:59:59.535991 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.535958    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzsx\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-kube-api-access-gkzsx\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:59.536178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.535992    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:59.536178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.536011    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:59.536178 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.536027    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b7d530-db61-4ec9-9647-deb92b22a82f-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:59.886250 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.886216    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9" event={"ID":"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1","Type":"ContainerStarted","Data":"7ff11ec02712baf2e045154af4b6028fd56e3a09b93b50cf7e04d69bdfb0154b"}
Nov 01 12:59:59.886436 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.886315    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 12:59:59.888027 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.888003    5725 generic.go:356] "Generic (PLEG): container finished" podID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerID="c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8" exitCode=0
Nov 01 12:59:59.888179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.888032    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" event={"ID":"c0b7d530-db61-4ec9-9647-deb92b22a82f","Type":"ContainerDied","Data":"c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8"}
Nov 01 12:59:59.888179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.888064    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"
Nov 01 12:59:59.888179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.888075    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8" event={"ID":"c0b7d530-db61-4ec9-9647-deb92b22a82f","Type":"ContainerDied","Data":"2f3497d67ec79e2693608d11a48929dae654623db80706f26c583caf8aa7178a"}
Nov 01 12:59:59.888179 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.888096    5725 scope.go:122] "RemoveContainer" containerID="c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8"
Nov 01 12:59:59.900212 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.900156    5725 scope.go:122] "RemoveContainer" containerID="c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2"
Nov 01 12:59:59.909952 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.909936    5725 scope.go:122] "RemoveContainer" containerID="c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8"
Nov 01 12:59:59.910226 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:59.910208    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8\": container with ID starting with c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8 not found: ID does not exist" containerID="c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8"
Nov 01 12:59:59.910275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.910234    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8"} err="failed to get container status \"c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8\": rpc error: code = NotFound desc = could not find container \"c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8\": container with ID starting with c2f5fcd7ba517e18e8475f3a93177440403a5c38fa2dfeb3d15e7f10ced010d8 not found: ID does not exist"
Nov 01 12:59:59.910275 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.910252    5725 scope.go:122] "RemoveContainer" containerID="c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2"
Nov 01 12:59:59.910464 ip-10-0-138-76 kubenswrapper[5725]: E1101 12:59:59.910445    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2\": container with ID starting with c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2 not found: ID does not exist" containerID="c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2"
Nov 01 12:59:59.910501 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.910471    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2"} err="failed to get container status \"c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2\": rpc error: code = NotFound desc = could not find container \"c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2\": container with ID starting with c137e672f5eead6f62665a80c0097d4473fb10559ad8bfadd541e69aaf0ed5e2 not found: ID does not exist"
Nov 01 12:59:59.999672 ip-10-0-138-76 kubenswrapper[5725]: I1101 12:59:59.999623    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9" podStartSLOduration=3.060755784 podStartE2EDuration="3.999609184s" totalImagesPullingTime="938.8534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:59:59.998275156 +0000 UTC m=+2241.505466373" watchObservedRunningTime="2026-11-01 12:59:59.999609184 +0000 UTC m=+2241.506800402"
Nov 01 13:00:00.107911 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.107875    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"]
Nov 01 13:00:00.111036 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.110986    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0cd09f-openshift-default-5658876955-v4gt8"]
Nov 01 13:00:00.171865 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.171786    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"]
Nov 01 13:00:00.172365 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.172350    5725 cpu_manager.go:414] "containerMap: removing container" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy"
Nov 01 13:00:00.172424 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.172366    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4cd30c-3bc0-4a37-b808-0ce18ca0a8c9" containerName="istio-proxy"
Nov 01 13:00:00.172424 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.172387    5725 cpu_manager.go:414] "containerMap: removing container" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy"
Nov 01 13:00:00.172424 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.172392    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy"
Nov 01 13:00:00.172424 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.172406    5725 cpu_manager.go:414] "containerMap: removing container" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy"
Nov 01 13:00:00.172424 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.172412    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy"
Nov 01 13:00:00.172602 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.172502    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy"
Nov 01 13:00:00.172602 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.172512    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" containerName="istio-proxy"
Nov 01 13:00:00.177336 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.177316    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:00.202720 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.202681    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"]
Nov 01 13:00:00.206962 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.206940    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"]
Nov 01 13:00:00.207110 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.207029    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:00.210836 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.210815    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:00.217598 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.217548    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"]
Nov 01 13:00:00.221929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.221909    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:00.233673 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.233637    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"]
Nov 01 13:00:00.252930 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.246175    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79488639-741b-4eae-a32c-6315c2c6fe55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n\" (UID: \"79488639-741b-4eae-a32c-6315c2c6fe55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:00.252930 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.246253    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0309900-e26d-42bf-8138-cd5cdcc28d36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6\" (UID: \"a0309900-e26d-42bf-8138-cd5cdcc28d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:00.252930 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.246306    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6635c843-311e-4f2f-a17c-a797b86cd473-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb\" (UID: \"6635c843-311e-4f2f-a17c-a797b86cd473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:00.252930 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.246441    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhv5\" (UniqueName: \"kubernetes.io/projected/79488639-741b-4eae-a32c-6315c2c6fe55-kube-api-access-7bhv5\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n\" (UID: \"79488639-741b-4eae-a32c-6315c2c6fe55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:00.252930 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.246721    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9q8\" (UniqueName: \"kubernetes.io/projected/6635c843-311e-4f2f-a17c-a797b86cd473-kube-api-access-rj9q8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb\" (UID: \"6635c843-311e-4f2f-a17c-a797b86cd473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:00.252930 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.246825    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrp6\" (UniqueName: \"kubernetes.io/projected/a0309900-e26d-42bf-8138-cd5cdcc28d36-kube-api-access-dlrp6\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6\" (UID: \"a0309900-e26d-42bf-8138-cd5cdcc28d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:00.272209 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.269561    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"]
Nov 01 13:00:00.274125 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.274091    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"]
Nov 01 13:00:00.322404 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.322364    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:00:00.330773 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.330740    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"]
Nov 01 13:00:00.348564 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.348527    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79488639-741b-4eae-a32c-6315c2c6fe55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n\" (UID: \"79488639-741b-4eae-a32c-6315c2c6fe55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:00.348754 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.348575    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0309900-e26d-42bf-8138-cd5cdcc28d36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6\" (UID: \"a0309900-e26d-42bf-8138-cd5cdcc28d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:00.348886 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.348853    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6635c843-311e-4f2f-a17c-a797b86cd473-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb\" (UID: \"6635c843-311e-4f2f-a17c-a797b86cd473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:00.349033 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.348928    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhv5\" (UniqueName: \"kubernetes.io/projected/79488639-741b-4eae-a32c-6315c2c6fe55-kube-api-access-7bhv5\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n\" (UID: \"79488639-741b-4eae-a32c-6315c2c6fe55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:00.349153 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.349132    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9q8\" (UniqueName: \"kubernetes.io/projected/6635c843-311e-4f2f-a17c-a797b86cd473-kube-api-access-rj9q8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb\" (UID: \"6635c843-311e-4f2f-a17c-a797b86cd473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:00.349238 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.349169    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gp2j\" (UniqueName: \"kubernetes.io/projected/c24d3559-6438-4932-b75a-6d5fb9825633-kube-api-access-5gp2j\") pod \"maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6\" (UID: \"c24d3559-6438-4932-b75a-6d5fb9825633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:00.349238 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.349201    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrp6\" (UniqueName: \"kubernetes.io/projected/a0309900-e26d-42bf-8138-cd5cdcc28d36-kube-api-access-dlrp6\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6\" (UID: \"a0309900-e26d-42bf-8138-cd5cdcc28d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:00.349368 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.349276    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24d3559-6438-4932-b75a-6d5fb9825633-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6\" (UID: \"c24d3559-6438-4932-b75a-6d5fb9825633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:00.349518 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.349481    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0309900-e26d-42bf-8138-cd5cdcc28d36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6\" (UID: \"a0309900-e26d-42bf-8138-cd5cdcc28d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:00.349518 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.349487    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79488639-741b-4eae-a32c-6315c2c6fe55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n\" (UID: \"79488639-741b-4eae-a32c-6315c2c6fe55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:00.349657 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.349534    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6635c843-311e-4f2f-a17c-a797b86cd473-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb\" (UID: \"6635c843-311e-4f2f-a17c-a797b86cd473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:00.403766 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.403729    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrp6\" (UniqueName: \"kubernetes.io/projected/a0309900-e26d-42bf-8138-cd5cdcc28d36-kube-api-access-dlrp6\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6\" (UID: \"a0309900-e26d-42bf-8138-cd5cdcc28d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:00.404155 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.403731    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhv5\" (UniqueName: \"kubernetes.io/projected/79488639-741b-4eae-a32c-6315c2c6fe55-kube-api-access-7bhv5\") pod \"maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n\" (UID: \"79488639-741b-4eae-a32c-6315c2c6fe55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:00.404155 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.403833    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9q8\" (UniqueName: \"kubernetes.io/projected/6635c843-311e-4f2f-a17c-a797b86cd473-kube-api-access-rj9q8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb\" (UID: \"6635c843-311e-4f2f-a17c-a797b86cd473\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:00.452809 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.452719    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gp2j\" (UniqueName: \"kubernetes.io/projected/c24d3559-6438-4932-b75a-6d5fb9825633-kube-api-access-5gp2j\") pod \"maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6\" (UID: \"c24d3559-6438-4932-b75a-6d5fb9825633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:00.452981 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.452808    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24d3559-6438-4932-b75a-6d5fb9825633-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6\" (UID: \"c24d3559-6438-4932-b75a-6d5fb9825633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:00.453497 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.453471    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24d3559-6438-4932-b75a-6d5fb9825633-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6\" (UID: \"c24d3559-6438-4932-b75a-6d5fb9825633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:00.492488 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.492455    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:00.509797 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.509765    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gp2j\" (UniqueName: \"kubernetes.io/projected/c24d3559-6438-4932-b75a-6d5fb9825633-kube-api-access-5gp2j\") pod \"maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6\" (UID: \"c24d3559-6438-4932-b75a-6d5fb9825633\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:00.522699 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.522670    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:00.542972 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.542935    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:00.560187 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.559603    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:00.683990 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.683501    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b7d530-db61-4ec9-9647-deb92b22a82f" path="/var/lib/kubelet/pods/c0b7d530-db61-4ec9-9647-deb92b22a82f/volumes"
Nov 01 13:00:00.892382 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.892351    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"]
Nov 01 13:00:00.899472 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:00:00.899443    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0309900_e26d_42bf_8138_cd5cdcc28d36.slice/crio-1d079f808ec40196a3563ac8770391554effe106c8c6ee35944465b443b47a4f WatchSource:0}: Error finding container 1d079f808ec40196a3563ac8770391554effe106c8c6ee35944465b443b47a4f: Status 404 returned error can't find the container with id 1d079f808ec40196a3563ac8770391554effe106c8c6ee35944465b443b47a4f
Nov 01 13:00:00.965608 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:00.965534    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"]
Nov 01 13:00:00.971786 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:00:00.971759    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79488639_741b_4eae_a32c_6315c2c6fe55.slice/crio-3f072bb3498d114fdd7a6ec7f050662f6a4e8d0f20f8d9975960628956d50a75 WatchSource:0}: Error finding container 3f072bb3498d114fdd7a6ec7f050662f6a4e8d0f20f8d9975960628956d50a75: Status 404 returned error can't find the container with id 3f072bb3498d114fdd7a6ec7f050662f6a4e8d0f20f8d9975960628956d50a75
Nov 01 13:00:01.006998 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.006969    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"]
Nov 01 13:00:01.013901 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:00:01.013851    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6635c843_311e_4f2f_a17c_a797b86cd473.slice/crio-6eff772e2bbe13143fe4e3ddbbb4bf820ef3c77cbe167d109dc098feb9d2affb WatchSource:0}: Error finding container 6eff772e2bbe13143fe4e3ddbbb4bf820ef3c77cbe167d109dc098feb9d2affb: Status 404 returned error can't find the container with id 6eff772e2bbe13143fe4e3ddbbb4bf820ef3c77cbe167d109dc098feb9d2affb
Nov 01 13:00:01.022689 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.022664    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"]
Nov 01 13:00:01.036276 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:01.036253    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:00:01.905759 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.905716    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" event={"ID":"a0309900-e26d-42bf-8138-cd5cdcc28d36","Type":"ContainerStarted","Data":"6a0c83973887e2554c9a665a8e5818249f8d81bcadbc15f64b334fb4cb84e874"}
Nov 01 13:00:01.906397 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.906365    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" event={"ID":"a0309900-e26d-42bf-8138-cd5cdcc28d36","Type":"ContainerStarted","Data":"1d079f808ec40196a3563ac8770391554effe106c8c6ee35944465b443b47a4f"}
Nov 01 13:00:01.907593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.907558    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" event={"ID":"79488639-741b-4eae-a32c-6315c2c6fe55","Type":"ContainerStarted","Data":"7efecfd253bf1b3c6af3b72e1176864e1accf086d9814fc93b8e59fe2be9e607"}
Nov 01 13:00:01.907722 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.907612    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" event={"ID":"79488639-741b-4eae-a32c-6315c2c6fe55","Type":"ContainerStarted","Data":"3f072bb3498d114fdd7a6ec7f050662f6a4e8d0f20f8d9975960628956d50a75"}
Nov 01 13:00:01.913743 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.913716    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" event={"ID":"c24d3559-6438-4932-b75a-6d5fb9825633","Type":"ContainerStarted","Data":"6cf25ef3fb4d450cb1441fd6a41138cebf6f24c93298f40638d12aff8c852263"}
Nov 01 13:00:01.913882 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.913763    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" event={"ID":"c24d3559-6438-4932-b75a-6d5fb9825633","Type":"ContainerStarted","Data":"e0d5ea086c1870c6a0c3112205e6057d42f4fbce253e736e0296acb58cfbc8f5"}
Nov 01 13:00:01.915418 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.915392    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb" event={"ID":"6635c843-311e-4f2f-a17c-a797b86cd473","Type":"ContainerStarted","Data":"f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a"}
Nov 01 13:00:01.915548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:01.915439    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb" event={"ID":"6635c843-311e-4f2f-a17c-a797b86cd473","Type":"ContainerStarted","Data":"6eff772e2bbe13143fe4e3ddbbb4bf820ef3c77cbe167d109dc098feb9d2affb"}
Nov 01 13:00:02.017480 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.017427    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" podStartSLOduration=2.017409387 podStartE2EDuration="2.017409387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:02.016473711 +0000 UTC m=+2243.523664933" watchObservedRunningTime="2026-11-01 13:00:02.017409387 +0000 UTC m=+2243.524600602"
Nov 01 13:00:02.127514 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.127402    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" podStartSLOduration=2.127383386 podStartE2EDuration="2.127383386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:02.123756728 +0000 UTC m=+2243.630947944" watchObservedRunningTime="2026-11-01 13:00:02.127383386 +0000 UTC m=+2243.634574604"
Nov 01 13:00:02.234221 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.234146    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb" podStartSLOduration=2.234124377 podStartE2EDuration="2.234124377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:02.231423932 +0000 UTC m=+2243.738615160" watchObservedRunningTime="2026-11-01 13:00:02.234124377 +0000 UTC m=+2243.741315598"
Nov 01 13:00:02.341104 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.341010    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" podStartSLOduration=2.34099644 podStartE2EDuration="2.34099644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:02.338779829 +0000 UTC m=+2243.845971047" watchObservedRunningTime="2026-11-01 13:00:02.34099644 +0000 UTC m=+2243.848187696"
Nov 01 13:00:02.637403 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.637361    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"]
Nov 01 13:00:02.924476 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.924382    5725 generic.go:356] "Generic (PLEG): container finished" podID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerID="6a0c83973887e2554c9a665a8e5818249f8d81bcadbc15f64b334fb4cb84e874" exitCode=60
Nov 01 13:00:02.924935 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.924478    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" event={"ID":"a0309900-e26d-42bf-8138-cd5cdcc28d36","Type":"ContainerDied","Data":"6a0c83973887e2554c9a665a8e5818249f8d81bcadbc15f64b334fb4cb84e874"}
Nov 01 13:00:02.924935 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.924744    5725 scope.go:122] "RemoveContainer" containerID="6a0c83973887e2554c9a665a8e5818249f8d81bcadbc15f64b334fb4cb84e874"
Nov 01 13:00:02.926384 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.926359    5725 generic.go:356] "Generic (PLEG): container finished" podID="79488639-741b-4eae-a32c-6315c2c6fe55" containerID="7efecfd253bf1b3c6af3b72e1176864e1accf086d9814fc93b8e59fe2be9e607" exitCode=60
Nov 01 13:00:02.926384 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.926383    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" event={"ID":"79488639-741b-4eae-a32c-6315c2c6fe55","Type":"ContainerDied","Data":"7efecfd253bf1b3c6af3b72e1176864e1accf086d9814fc93b8e59fe2be9e607"}
Nov 01 13:00:02.926708 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.926684    5725 scope.go:122] "RemoveContainer" containerID="7efecfd253bf1b3c6af3b72e1176864e1accf086d9814fc93b8e59fe2be9e607"
Nov 01 13:00:02.929948 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.929910    5725 generic.go:356] "Generic (PLEG): container finished" podID="c24d3559-6438-4932-b75a-6d5fb9825633" containerID="6cf25ef3fb4d450cb1441fd6a41138cebf6f24c93298f40638d12aff8c852263" exitCode=60
Nov 01 13:00:02.930322 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.930286    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" event={"ID":"c24d3559-6438-4932-b75a-6d5fb9825633","Type":"ContainerDied","Data":"6cf25ef3fb4d450cb1441fd6a41138cebf6f24c93298f40638d12aff8c852263"}
Nov 01 13:00:02.930753 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.930733    5725 scope.go:122] "RemoveContainer" containerID="6cf25ef3fb4d450cb1441fd6a41138cebf6f24c93298f40638d12aff8c852263"
Nov 01 13:00:02.932442 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.932417    5725 generic.go:356] "Generic (PLEG): container finished" podID="6635c843-311e-4f2f-a17c-a797b86cd473" containerID="f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a" exitCode=60
Nov 01 13:00:02.932800 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.932573    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb" event={"ID":"6635c843-311e-4f2f-a17c-a797b86cd473","Type":"ContainerDied","Data":"f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a"}
Nov 01 13:00:02.932894 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:02.932690    5725 scope.go:122] "RemoveContainer" containerID="f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a"
Nov 01 13:00:03.736036 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:03.736001    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"]
Nov 01 13:00:03.736256 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:03.736211    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk" podUID="58501f10-37c4-4d76-acda-ae7131ea52fc" containerName="maas-api" containerID="cri-o://5de8e0af67515248588d1fd543ec02de818f445e259bd13ab9a63401074f793a" gracePeriod=30
Nov 01 13:00:03.954575 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:03.954535    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb" event={"ID":"6635c843-311e-4f2f-a17c-a797b86cd473","Type":"ContainerStarted","Data":"5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042"}
Nov 01 13:00:03.955018 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:03.954764    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb" podUID="6635c843-311e-4f2f-a17c-a797b86cd473" containerName="cleanup" containerID="cri-o://5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042" gracePeriod=30
Nov 01 13:00:03.964532 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:03.964494    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" event={"ID":"a0309900-e26d-42bf-8138-cd5cdcc28d36","Type":"ContainerStarted","Data":"8a6f61bfc8ede4079f3d7a64dfef7b1816ddc4a54f5f93321f02dd6fc3f7d297"}
Nov 01 13:00:03.968089 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:03.967793    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" event={"ID":"79488639-741b-4eae-a32c-6315c2c6fe55","Type":"ContainerStarted","Data":"7748363ff261f41b08a981e057c3f299fe64298db3641900289e0742709eff90"}
Nov 01 13:00:03.978638 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:03.978601    5725 generic.go:356] "Generic (PLEG): container finished" podID="58501f10-37c4-4d76-acda-ae7131ea52fc" containerID="5de8e0af67515248588d1fd543ec02de818f445e259bd13ab9a63401074f793a" exitCode=0
Nov 01 13:00:03.978900 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:03.978879    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk" event={"ID":"58501f10-37c4-4d76-acda-ae7131ea52fc","Type":"ContainerDied","Data":"5de8e0af67515248588d1fd543ec02de818f445e259bd13ab9a63401074f793a"}
Nov 01 13:00:03.983583 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:03.983550    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" event={"ID":"c24d3559-6438-4932-b75a-6d5fb9825633","Type":"ContainerStarted","Data":"ffd36fe099effb6e43844fa38b531091aebd9e8f8cd436a27590fd7fe19e7bb4"}
Nov 01 13:00:04.149401 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.149369    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 13:00:04.213633 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.213596    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls\") pod \"58501f10-37c4-4d76-acda-ae7131ea52fc\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") "
Nov 01 13:00:04.213829 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.213728    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58501f10-37c4-4d76-acda-ae7131ea52fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58501f10-37c4-4d76-acda-ae7131ea52fc-openshift-service-ca\") pod \"58501f10-37c4-4d76-acda-ae7131ea52fc\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") "
Nov 01 13:00:04.213829 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.213799    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58501f10-37c4-4d76-acda-ae7131ea52fc-kube-api-access-z9n2x\" (UniqueName: \"kubernetes.io/projected/58501f10-37c4-4d76-acda-ae7131ea52fc-kube-api-access-z9n2x\") pod \"58501f10-37c4-4d76-acda-ae7131ea52fc\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") "
Nov 01 13:00:04.213939 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.213834    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-metrics-certs\") pod \"58501f10-37c4-4d76-acda-ae7131ea52fc\" (UID: \"58501f10-37c4-4d76-acda-ae7131ea52fc\") "
Nov 01 13:00:04.214163 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.214129    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58501f10-37c4-4d76-acda-ae7131ea52fc-openshift-service-ca" pod "58501f10-37c4-4d76-acda-ae7131ea52fc" (UID: "58501f10-37c4-4d76-acda-ae7131ea52fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:04.216065 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.216025    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-metrics-certs" pod "58501f10-37c4-4d76-acda-ae7131ea52fc" (UID: "58501f10-37c4-4d76-acda-ae7131ea52fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:04.216364 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.216346    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls" pod "58501f10-37c4-4d76-acda-ae7131ea52fc" (UID: "58501f10-37c4-4d76-acda-ae7131ea52fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:04.216430 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.216372    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58501f10-37c4-4d76-acda-ae7131ea52fc-kube-api-access-z9n2x" pod "58501f10-37c4-4d76-acda-ae7131ea52fc" (UID: "58501f10-37c4-4d76-acda-ae7131ea52fc"). InnerVolumeSpecName "kube-api-access-z9n2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:04.315266 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.315178    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58501f10-37c4-4d76-acda-ae7131ea52fc-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:04.315266 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.315210    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9n2x\" (UniqueName: \"kubernetes.io/projected/58501f10-37c4-4d76-acda-ae7131ea52fc-kube-api-access-z9n2x\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:04.315266 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.315221    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:04.315266 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.315230    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58501f10-37c4-4d76-acda-ae7131ea52fc-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:04.995029 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.994991    5725 generic.go:356] "Generic (PLEG): container finished" podID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerID="8a6f61bfc8ede4079f3d7a64dfef7b1816ddc4a54f5f93321f02dd6fc3f7d297" exitCode=60
Nov 01 13:00:04.995568 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.995078    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" event={"ID":"a0309900-e26d-42bf-8138-cd5cdcc28d36","Type":"ContainerDied","Data":"8a6f61bfc8ede4079f3d7a64dfef7b1816ddc4a54f5f93321f02dd6fc3f7d297"}
Nov 01 13:00:04.995568 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.995158    5725 scope.go:122] "RemoveContainer" containerID="6a0c83973887e2554c9a665a8e5818249f8d81bcadbc15f64b334fb4cb84e874"
Nov 01 13:00:04.995568 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.995435    5725 scope.go:122] "RemoveContainer" containerID="8a6f61bfc8ede4079f3d7a64dfef7b1816ddc4a54f5f93321f02dd6fc3f7d297"
Nov 01 13:00:04.995828 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:04.995802    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6_odh-ai-gateway-infra(a0309900-e26d-42bf-8138-cd5cdcc28d36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36"
Nov 01 13:00:04.997451 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.997427    5725 generic.go:356] "Generic (PLEG): container finished" podID="79488639-741b-4eae-a32c-6315c2c6fe55" containerID="7748363ff261f41b08a981e057c3f299fe64298db3641900289e0742709eff90" exitCode=60
Nov 01 13:00:04.997576 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.997511    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" event={"ID":"79488639-741b-4eae-a32c-6315c2c6fe55","Type":"ContainerDied","Data":"7748363ff261f41b08a981e057c3f299fe64298db3641900289e0742709eff90"}
Nov 01 13:00:04.997752 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.997736    5725 scope.go:122] "RemoveContainer" containerID="7748363ff261f41b08a981e057c3f299fe64298db3641900289e0742709eff90"
Nov 01 13:00:04.997993 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:04.997974    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n_odh-ai-gateway-infra(79488639-741b-4eae-a32c-6315c2c6fe55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" podUID="79488639-741b-4eae-a32c-6315c2c6fe55"
Nov 01 13:00:04.999081 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.999033    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"
Nov 01 13:00:04.999081 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:04.999038    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk" event={"ID":"58501f10-37c4-4d76-acda-ae7131ea52fc","Type":"ContainerDied","Data":"dc55b2fe0dbc40b43029c9d28cd5a352846f8e1db454882efeb184bfe017e162"}
Nov 01 13:00:05.001505 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:05.001484    5725 generic.go:356] "Generic (PLEG): container finished" podID="c24d3559-6438-4932-b75a-6d5fb9825633" containerID="ffd36fe099effb6e43844fa38b531091aebd9e8f8cd436a27590fd7fe19e7bb4" exitCode=60
Nov 01 13:00:05.001613 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:05.001544    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" event={"ID":"c24d3559-6438-4932-b75a-6d5fb9825633","Type":"ContainerDied","Data":"ffd36fe099effb6e43844fa38b531091aebd9e8f8cd436a27590fd7fe19e7bb4"}
Nov 01 13:00:05.001903 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:05.001886    5725 scope.go:122] "RemoveContainer" containerID="ffd36fe099effb6e43844fa38b531091aebd9e8f8cd436a27590fd7fe19e7bb4"
Nov 01 13:00:05.002185 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:05.002163    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6_odh-ai-gateway-infra(c24d3559-6438-4932-b75a-6d5fb9825633)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" podUID="c24d3559-6438-4932-b75a-6d5fb9825633"
Nov 01 13:00:05.012790 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:05.012751    5725 scope.go:122] "RemoveContainer" containerID="7efecfd253bf1b3c6af3b72e1176864e1accf086d9814fc93b8e59fe2be9e607"
Nov 01 13:00:05.035799 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:05.035773    5725 scope.go:122] "RemoveContainer" containerID="5de8e0af67515248588d1fd543ec02de818f445e259bd13ab9a63401074f793a"
Nov 01 13:00:05.046961 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:05.046941    5725 scope.go:122] "RemoveContainer" containerID="6cf25ef3fb4d450cb1441fd6a41138cebf6f24c93298f40638d12aff8c852263"
Nov 01 13:00:05.209937 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:05.209898    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"]
Nov 01 13:00:05.218125 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:05.218088    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-94c13a-76445b5bfb-lftjk"]
Nov 01 13:00:05.903336 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:05.903307    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 13:00:06.010138 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:06.010104    5725 scope.go:122] "RemoveContainer" containerID="8a6f61bfc8ede4079f3d7a64dfef7b1816ddc4a54f5f93321f02dd6fc3f7d297"
Nov 01 13:00:06.010620 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:06.010430    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6_odh-ai-gateway-infra(a0309900-e26d-42bf-8138-cd5cdcc28d36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36"
Nov 01 13:00:06.011983 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:06.011954    5725 scope.go:122] "RemoveContainer" containerID="7748363ff261f41b08a981e057c3f299fe64298db3641900289e0742709eff90"
Nov 01 13:00:06.012287 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:06.012262    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n_odh-ai-gateway-infra(79488639-741b-4eae-a32c-6315c2c6fe55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" podUID="79488639-741b-4eae-a32c-6315c2c6fe55"
Nov 01 13:00:06.028155 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:06.028127    5725 scope.go:122] "RemoveContainer" containerID="ffd36fe099effb6e43844fa38b531091aebd9e8f8cd436a27590fd7fe19e7bb4"
Nov 01 13:00:06.028456 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:06.028432    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6_odh-ai-gateway-infra(c24d3559-6438-4932-b75a-6d5fb9825633)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" podUID="c24d3559-6438-4932-b75a-6d5fb9825633"
Nov 01 13:00:06.388762 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:06.388723    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"]
Nov 01 13:00:06.394695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:06.394666    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94c13a-sn9ds"]
Nov 01 13:00:06.616925 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:06.616894    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58501f10-37c4-4d76-acda-ae7131ea52fc" path="/var/lib/kubelet/pods/58501f10-37c4-4d76-acda-ae7131ea52fc/volumes"
Nov 01 13:00:06.617405 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:06.617389    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff7b9fe-1f97-42c8-8ee1-c6f79edc856c" path="/var/lib/kubelet/pods/cff7b9fe-1f97-42c8-8ee1-c6f79edc856c/volumes"
Nov 01 13:00:14.545644 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:14.545594    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main" containerID="cri-o://4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730" gracePeriod=45
Nov 01 13:00:14.994484 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:14.994451    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 13:00:15.104069 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.103470    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72055ebb-472b-43c6-8d29-e7a870069f31-tls-certs\" (UniqueName: \"kubernetes.io/secret/72055ebb-472b-43c6-8d29-e7a870069f31-tls-certs\") pod \"72055ebb-472b-43c6-8d29-e7a870069f31\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") "
Nov 01 13:00:15.104069 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.103590    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72055ebb-472b-43c6-8d29-e7a870069f31-kube-api-access-9d7nn\" (UniqueName: \"kubernetes.io/projected/72055ebb-472b-43c6-8d29-e7a870069f31-kube-api-access-9d7nn\") pod \"72055ebb-472b-43c6-8d29-e7a870069f31\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") "
Nov 01 13:00:15.104069 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.103622    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-tmp-dir\") pod \"72055ebb-472b-43c6-8d29-e7a870069f31\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") "
Nov 01 13:00:15.104069 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.103759    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-dshm\") pod \"72055ebb-472b-43c6-8d29-e7a870069f31\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") "
Nov 01 13:00:15.104069 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.103807    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-model-cache\") pod \"72055ebb-472b-43c6-8d29-e7a870069f31\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") "
Nov 01 13:00:15.104069 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.103841    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-home\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-home\") pod \"72055ebb-472b-43c6-8d29-e7a870069f31\" (UID: \"72055ebb-472b-43c6-8d29-e7a870069f31\") "
Nov 01 13:00:15.110104 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.109715    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-home" pod "72055ebb-472b-43c6-8d29-e7a870069f31" (UID: "72055ebb-472b-43c6-8d29-e7a870069f31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:15.111232 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.111198    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-tmp-dir" pod "72055ebb-472b-43c6-8d29-e7a870069f31" (UID: "72055ebb-472b-43c6-8d29-e7a870069f31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:15.112436 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.112044    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-model-cache" pod "72055ebb-472b-43c6-8d29-e7a870069f31" (UID: "72055ebb-472b-43c6-8d29-e7a870069f31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:15.115603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.115484    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-dshm" pod "72055ebb-472b-43c6-8d29-e7a870069f31" (UID: "72055ebb-472b-43c6-8d29-e7a870069f31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:15.115709 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.115630    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72055ebb-472b-43c6-8d29-e7a870069f31-kube-api-access-9d7nn" pod "72055ebb-472b-43c6-8d29-e7a870069f31" (UID: "72055ebb-472b-43c6-8d29-e7a870069f31"). InnerVolumeSpecName "kube-api-access-9d7nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:15.120659 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.120628    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72055ebb-472b-43c6-8d29-e7a870069f31-tls-certs" pod "72055ebb-472b-43c6-8d29-e7a870069f31" (UID: "72055ebb-472b-43c6-8d29-e7a870069f31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:15.128242 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.125966    5725 generic.go:356] "Generic (PLEG): container finished" podID="72055ebb-472b-43c6-8d29-e7a870069f31" containerID="4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730" exitCode=0
Nov 01 13:00:15.128242 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.126175    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" event={"ID":"72055ebb-472b-43c6-8d29-e7a870069f31","Type":"ContainerDied","Data":"4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730"}
Nov 01 13:00:15.128242 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.126214    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq" event={"ID":"72055ebb-472b-43c6-8d29-e7a870069f31","Type":"ContainerDied","Data":"4687da12ca171195d7bf8d805bc0afed609a7c5145cdf6c72bc0a24b81770a42"}
Nov 01 13:00:15.128242 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.126234    5725 scope.go:122] "RemoveContainer" containerID="4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730"
Nov 01 13:00:15.128242 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.126398    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"
Nov 01 13:00:15.171829 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.171802    5725 scope.go:122] "RemoveContainer" containerID="4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730"
Nov 01 13:00:15.172522 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:15.172495    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730\": container with ID starting with 4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730 not found: ID does not exist" containerID="4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730"
Nov 01 13:00:15.172642 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.172527    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730"} err="failed to get container status \"4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730\": rpc error: code = NotFound desc = could not find container \"4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730\": container with ID starting with 4f284b88aabdb81ea115f24414e3498771a691da986fcb34c4b48a1261f99730 not found: ID does not exist"
Nov 01 13:00:15.208042 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.207976    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.208042 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.208004    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.208042 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.208025    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.208042 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.208034    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72055ebb-472b-43c6-8d29-e7a870069f31-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.208042 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.208042    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d7nn\" (UniqueName: \"kubernetes.io/projected/72055ebb-472b-43c6-8d29-e7a870069f31-kube-api-access-9d7nn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.208324 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.208073    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72055ebb-472b-43c6-8d29-e7a870069f31-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.324543 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.324510    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"]
Nov 01 13:00:15.357159 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.357126    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/premium-w1-5b1bd4-kserve-85ff69ccd9-2l4xq"]
Nov 01 13:00:15.611331 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.609654    5725 scope.go:122] "RemoveContainer" containerID="8a6f61bfc8ede4079f3d7a64dfef7b1816ddc4a54f5f93321f02dd6fc3f7d297"
Nov 01 13:00:15.611331 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.610348    5725 scope.go:122] "RemoveContainer" containerID="7748363ff261f41b08a981e057c3f299fe64298db3641900289e0742709eff90"
Nov 01 13:00:15.700099 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.700014    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"]
Nov 01 13:00:15.701498 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.701476    5725 cpu_manager.go:414] "containerMap: removing container" podUID="58501f10-37c4-4d76-acda-ae7131ea52fc" containerName="maas-api"
Nov 01 13:00:15.701498 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.701497    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58501f10-37c4-4d76-acda-ae7131ea52fc" containerName="maas-api"
Nov 01 13:00:15.701649 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.701513    5725 cpu_manager.go:414] "containerMap: removing container" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main"
Nov 01 13:00:15.701649 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.701521    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main"
Nov 01 13:00:15.701649 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.701648    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" containerName="main"
Nov 01 13:00:15.701750 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.701669    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="58501f10-37c4-4d76-acda-ae7131ea52fc" containerName="maas-api"
Nov 01 13:00:15.706684 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.706669    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:15.761086 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.760935    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:00:15.762843 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.762812    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"]
Nov 01 13:00:15.814913 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.814870    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:15.815176 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.815157    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdjhg\" (UniqueName: \"kubernetes.io/projected/8f69a0a6-7b30-47dc-b148-8e5700613790-kube-api-access-cdjhg\") pod \"maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:15.815311 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.815289    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:15.917081 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.916964    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdjhg\" (UniqueName: \"kubernetes.io/projected/8f69a0a6-7b30-47dc-b148-8e5700613790-kube-api-access-cdjhg\") pod \"maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:15.917081 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.917030    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:15.917356 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.917122    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:15.917657 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.917627    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:15.919911 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.919887    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:15.975566 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:15.975536    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdjhg\" (UniqueName: \"kubernetes.io/projected/8f69a0a6-7b30-47dc-b148-8e5700613790-kube-api-access-cdjhg\") pod \"maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:16.020981 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:16.020603    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:16.458372 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:16.458290    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"]
Nov 01 13:00:16.617645 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:16.617607    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72055ebb-472b-43c6-8d29-e7a870069f31" path="/var/lib/kubelet/pods/72055ebb-472b-43c6-8d29-e7a870069f31/volumes"
Nov 01 13:00:17.003538 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.003453    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"]
Nov 01 13:00:17.012404 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.012377    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.066168 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.064560    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"]
Nov 01 13:00:17.163845 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.163780    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-112d13a5-glqgl\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.164017 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.163938    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5thq\" (UniqueName: \"kubernetes.io/projected/7fecb164-8181-44b0-9264-8d18ca7f1f60-kube-api-access-t5thq\") pod \"maas-api-revoke-sub-112d13a5-glqgl\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.164017 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.164007    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-112d13a5-glqgl\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.183466 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.183435    5725 generic.go:356] "Generic (PLEG): container finished" podID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerID="ea573d4ef2d5146f5b669f46ae325f35debe39e6dcdec32db1312e86625991dc" exitCode=60
Nov 01 13:00:17.183640 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.183510    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" event={"ID":"a0309900-e26d-42bf-8138-cd5cdcc28d36","Type":"ContainerDied","Data":"ea573d4ef2d5146f5b669f46ae325f35debe39e6dcdec32db1312e86625991dc"}
Nov 01 13:00:17.183640 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.183579    5725 scope.go:122] "RemoveContainer" containerID="8a6f61bfc8ede4079f3d7a64dfef7b1816ddc4a54f5f93321f02dd6fc3f7d297"
Nov 01 13:00:17.184193 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.184174    5725 scope.go:122] "RemoveContainer" containerID="ea573d4ef2d5146f5b669f46ae325f35debe39e6dcdec32db1312e86625991dc"
Nov 01 13:00:17.184427 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:17.184409    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6_odh-ai-gateway-infra(a0309900-e26d-42bf-8138-cd5cdcc28d36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36"
Nov 01 13:00:17.185595 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.185575    5725 generic.go:356] "Generic (PLEG): container finished" podID="79488639-741b-4eae-a32c-6315c2c6fe55" containerID="fe63d60c632473910eb8504bd281399cbb34264f6709402937c368b489d86101" exitCode=60
Nov 01 13:00:17.185703 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.185653    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" event={"ID":"79488639-741b-4eae-a32c-6315c2c6fe55","Type":"ContainerDied","Data":"fe63d60c632473910eb8504bd281399cbb34264f6709402937c368b489d86101"}
Nov 01 13:00:17.186016 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.185991    5725 scope.go:122] "RemoveContainer" containerID="fe63d60c632473910eb8504bd281399cbb34264f6709402937c368b489d86101"
Nov 01 13:00:17.186273 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:17.186253    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n_odh-ai-gateway-infra(79488639-741b-4eae-a32c-6315c2c6fe55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" podUID="79488639-741b-4eae-a32c-6315c2c6fe55"
Nov 01 13:00:17.187373 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.187349    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr" event={"ID":"8f69a0a6-7b30-47dc-b148-8e5700613790","Type":"ContainerStarted","Data":"7be4fcedfc2959832625ab8207ea4d9ef18b1eac1de95d243c534dd629c8c8d6"}
Nov 01 13:00:17.187442 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.187395    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr" event={"ID":"8f69a0a6-7b30-47dc-b148-8e5700613790","Type":"ContainerStarted","Data":"b1bf072572deffb449a0b5faa7029f0815aad7ac652d6ca270b6739a4564b126"}
Nov 01 13:00:17.199028 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.198442    5725 scope.go:122] "RemoveContainer" containerID="7748363ff261f41b08a981e057c3f299fe64298db3641900289e0742709eff90"
Nov 01 13:00:17.264767 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.264690    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-112d13a5-glqgl\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.264767 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.264744    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5thq\" (UniqueName: \"kubernetes.io/projected/7fecb164-8181-44b0-9264-8d18ca7f1f60-kube-api-access-t5thq\") pod \"maas-api-revoke-sub-112d13a5-glqgl\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.264995 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.264783    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-112d13a5-glqgl\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.265532 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.265503    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-112d13a5-glqgl\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.267528 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.267505    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-112d13a5-glqgl\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.322488 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.322461    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5thq\" (UniqueName: \"kubernetes.io/projected/7fecb164-8181-44b0-9264-8d18ca7f1f60-kube-api-access-t5thq\") pod \"maas-api-revoke-sub-112d13a5-glqgl\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.365270 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.365237    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:17.415276 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.415212    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr" podStartSLOduration=2.415193514 podStartE2EDuration="2.415193514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:17.411489304 +0000 UTC m=+2258.918680523" watchObservedRunningTime="2026-11-01 13:00:17.415193514 +0000 UTC m=+2258.922384732"
Nov 01 13:00:17.609884 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.609856    5725 scope.go:122] "RemoveContainer" containerID="ffd36fe099effb6e43844fa38b531091aebd9e8f8cd436a27590fd7fe19e7bb4"
Nov 01 13:00:17.756247 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:17.756215    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"]
Nov 01 13:00:18.197252 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.197204    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl" event={"ID":"7fecb164-8181-44b0-9264-8d18ca7f1f60","Type":"ContainerStarted","Data":"73482b15b0f7beaca1989fbc322d392de476c655316d46acc21139a8211457db"}
Nov 01 13:00:18.197455 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.197275    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl" event={"ID":"7fecb164-8181-44b0-9264-8d18ca7f1f60","Type":"ContainerStarted","Data":"a8c584927098c854b83a0ce88fe729390bb15fd5cf6442f62b013e610eb2ac70"}
Nov 01 13:00:18.202923 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.202894    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" event={"ID":"c24d3559-6438-4932-b75a-6d5fb9825633","Type":"ContainerStarted","Data":"811a4df44ea7d8700719f668b06d25d5f2358b50702331a4217d3269bf8245ee"}
Nov 01 13:00:18.204405 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.204376    5725 generic.go:356] "Generic (PLEG): container finished" podID="8f69a0a6-7b30-47dc-b148-8e5700613790" containerID="7be4fcedfc2959832625ab8207ea4d9ef18b1eac1de95d243c534dd629c8c8d6" exitCode=0
Nov 01 13:00:18.204520 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.204449    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr" event={"ID":"8f69a0a6-7b30-47dc-b148-8e5700613790","Type":"ContainerDied","Data":"7be4fcedfc2959832625ab8207ea4d9ef18b1eac1de95d243c534dd629c8c8d6"}
Nov 01 13:00:18.306375 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.306324    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"]
Nov 01 13:00:18.309217 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.309138    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl" podStartSLOduration=2.309124683 podStartE2EDuration="2.309124683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:18.306356964 +0000 UTC m=+2259.813548181" watchObservedRunningTime="2026-11-01 13:00:18.309124683 +0000 UTC m=+2259.816315949"
Nov 01 13:00:18.524227 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.524200    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:18.564155 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.564116    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"]
Nov 01 13:00:18.686694 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.686666    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0309900-e26d-42bf-8138-cd5cdcc28d36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0309900-e26d-42bf-8138-cd5cdcc28d36-maas-api-service-ca\") pod \"a0309900-e26d-42bf-8138-cd5cdcc28d36\" (UID: \"a0309900-e26d-42bf-8138-cd5cdcc28d36\") "
Nov 01 13:00:18.687232 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.687127    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0309900-e26d-42bf-8138-cd5cdcc28d36-maas-api-service-ca" pod "a0309900-e26d-42bf-8138-cd5cdcc28d36" (UID: "a0309900-e26d-42bf-8138-cd5cdcc28d36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:18.687398 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.687381    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0309900-e26d-42bf-8138-cd5cdcc28d36-kube-api-access-dlrp6\" (UniqueName: \"kubernetes.io/projected/a0309900-e26d-42bf-8138-cd5cdcc28d36-kube-api-access-dlrp6\") pod \"a0309900-e26d-42bf-8138-cd5cdcc28d36\" (UID: \"a0309900-e26d-42bf-8138-cd5cdcc28d36\") "
Nov 01 13:00:18.687786 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.687771    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0309900-e26d-42bf-8138-cd5cdcc28d36-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:18.750107 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.747212    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0309900-e26d-42bf-8138-cd5cdcc28d36-kube-api-access-dlrp6" pod "a0309900-e26d-42bf-8138-cd5cdcc28d36" (UID: "a0309900-e26d-42bf-8138-cd5cdcc28d36"). InnerVolumeSpecName "kube-api-access-dlrp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:18.788429 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.788392    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrp6\" (UniqueName: \"kubernetes.io/projected/a0309900-e26d-42bf-8138-cd5cdcc28d36-kube-api-access-dlrp6\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:18.946288 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.946251    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:18.990244 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.990210    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79488639-741b-4eae-a32c-6315c2c6fe55-kube-api-access-7bhv5\" (UniqueName: \"kubernetes.io/projected/79488639-741b-4eae-a32c-6315c2c6fe55-kube-api-access-7bhv5\") pod \"79488639-741b-4eae-a32c-6315c2c6fe55\" (UID: \"79488639-741b-4eae-a32c-6315c2c6fe55\") "
Nov 01 13:00:18.990417 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.990292    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79488639-741b-4eae-a32c-6315c2c6fe55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79488639-741b-4eae-a32c-6315c2c6fe55-maas-api-service-ca\") pod \"79488639-741b-4eae-a32c-6315c2c6fe55\" (UID: \"79488639-741b-4eae-a32c-6315c2c6fe55\") "
Nov 01 13:00:18.990761 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.990730    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79488639-741b-4eae-a32c-6315c2c6fe55-maas-api-service-ca" pod "79488639-741b-4eae-a32c-6315c2c6fe55" (UID: "79488639-741b-4eae-a32c-6315c2c6fe55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:18.992406 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:18.992387    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79488639-741b-4eae-a32c-6315c2c6fe55-kube-api-access-7bhv5" pod "79488639-741b-4eae-a32c-6315c2c6fe55" (UID: "79488639-741b-4eae-a32c-6315c2c6fe55"). InnerVolumeSpecName "kube-api-access-7bhv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:19.092279 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.092184    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79488639-741b-4eae-a32c-6315c2c6fe55-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:19.092279 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.092212    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhv5\" (UniqueName: \"kubernetes.io/projected/79488639-741b-4eae-a32c-6315c2c6fe55-kube-api-access-7bhv5\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:19.215191 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.215156    5725 generic.go:356] "Generic (PLEG): container finished" podID="c24d3559-6438-4932-b75a-6d5fb9825633" containerID="811a4df44ea7d8700719f668b06d25d5f2358b50702331a4217d3269bf8245ee" exitCode=60
Nov 01 13:00:19.215413 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.215235    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" event={"ID":"c24d3559-6438-4932-b75a-6d5fb9825633","Type":"ContainerDied","Data":"811a4df44ea7d8700719f668b06d25d5f2358b50702331a4217d3269bf8245ee"}
Nov 01 13:00:19.215413 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.215312    5725 scope.go:122] "RemoveContainer" containerID="ffd36fe099effb6e43844fa38b531091aebd9e8f8cd436a27590fd7fe19e7bb4"
Nov 01 13:00:19.215732 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.215712    5725 scope.go:122] "RemoveContainer" containerID="811a4df44ea7d8700719f668b06d25d5f2358b50702331a4217d3269bf8245ee"
Nov 01 13:00:19.216006 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:19.215988    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6_odh-ai-gateway-infra(c24d3559-6438-4932-b75a-6d5fb9825633)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" podUID="c24d3559-6438-4932-b75a-6d5fb9825633"
Nov 01 13:00:19.217767 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.217740    5725 generic.go:356] "Generic (PLEG): container finished" podID="7fecb164-8181-44b0-9264-8d18ca7f1f60" containerID="73482b15b0f7beaca1989fbc322d392de476c655316d46acc21139a8211457db" exitCode=0
Nov 01 13:00:19.217869 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.217815    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl" event={"ID":"7fecb164-8181-44b0-9264-8d18ca7f1f60","Type":"ContainerDied","Data":"73482b15b0f7beaca1989fbc322d392de476c655316d46acc21139a8211457db"}
Nov 01 13:00:19.219379 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.219352    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"
Nov 01 13:00:19.219379 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.219358    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6" event={"ID":"a0309900-e26d-42bf-8138-cd5cdcc28d36","Type":"ContainerDied","Data":"1d079f808ec40196a3563ac8770391554effe106c8c6ee35944465b443b47a4f"}
Nov 01 13:00:19.221327 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.221301    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"
Nov 01 13:00:19.238541 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.229965    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n" event={"ID":"79488639-741b-4eae-a32c-6315c2c6fe55","Type":"ContainerDied","Data":"3f072bb3498d114fdd7a6ec7f050662f6a4e8d0f20f8d9975960628956d50a75"}
Nov 01 13:00:19.243343 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.243316    5725 scope.go:122] "RemoveContainer" containerID="ea573d4ef2d5146f5b669f46ae325f35debe39e6dcdec32db1312e86625991dc"
Nov 01 13:00:19.288741 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.288392    5725 scope.go:122] "RemoveContainer" containerID="fe63d60c632473910eb8504bd281399cbb34264f6709402937c368b489d86101"
Nov 01 13:00:19.450330 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.450300    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:19.496246 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.496212    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-xks-ca\") pod \"8f69a0a6-7b30-47dc-b148-8e5700613790\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") "
Nov 01 13:00:19.496468 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.496283    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-openshift-ca\") pod \"8f69a0a6-7b30-47dc-b148-8e5700613790\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") "
Nov 01 13:00:19.496468 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.496422    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f69a0a6-7b30-47dc-b148-8e5700613790-kube-api-access-cdjhg\" (UniqueName: \"kubernetes.io/projected/8f69a0a6-7b30-47dc-b148-8e5700613790-kube-api-access-cdjhg\") pod \"8f69a0a6-7b30-47dc-b148-8e5700613790\" (UID: \"8f69a0a6-7b30-47dc-b148-8e5700613790\") "
Nov 01 13:00:19.496650 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.496626    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-openshift-ca" pod "8f69a0a6-7b30-47dc-b148-8e5700613790" (UID: "8f69a0a6-7b30-47dc-b148-8e5700613790"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:19.496800 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.496775    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:19.498681 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.498658    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-xks-ca" pod "8f69a0a6-7b30-47dc-b148-8e5700613790" (UID: "8f69a0a6-7b30-47dc-b148-8e5700613790"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:19.498779 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.498715    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f69a0a6-7b30-47dc-b148-8e5700613790-kube-api-access-cdjhg" pod "8f69a0a6-7b30-47dc-b148-8e5700613790" (UID: "8f69a0a6-7b30-47dc-b148-8e5700613790"). InnerVolumeSpecName "kube-api-access-cdjhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:19.540524 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.540492    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"]
Nov 01 13:00:19.547314 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.547285    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"]
Nov 01 13:00:19.551826 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.551800    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-06fd6b-29892300-89t2n"]
Nov 01 13:00:19.598151 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.598117    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdjhg\" (UniqueName: \"kubernetes.io/projected/8f69a0a6-7b30-47dc-b148-8e5700613790-kube-api-access-cdjhg\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:19.598151 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.598149    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f69a0a6-7b30-47dc-b148-8e5700613790-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:19.651223 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.651186    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"]
Nov 01 13:00:19.654261 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:19.654232    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b1bd4-29892300-rqwg6"]
Nov 01 13:00:20.234220 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.234190    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"
Nov 01 13:00:20.234626 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.234189    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr" event={"ID":"8f69a0a6-7b30-47dc-b148-8e5700613790","Type":"ContainerDied","Data":"b1bf072572deffb449a0b5faa7029f0815aad7ac652d6ca270b6739a4564b126"}
Nov 01 13:00:20.234626 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.234321    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1bf072572deffb449a0b5faa7029f0815aad7ac652d6ca270b6739a4564b126"
Nov 01 13:00:20.414593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.414565    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:20.472190 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.472162    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:20.510247 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.510164    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fecb164-8181-44b0-9264-8d18ca7f1f60-kube-api-access-t5thq\" (UniqueName: \"kubernetes.io/projected/7fecb164-8181-44b0-9264-8d18ca7f1f60-kube-api-access-t5thq\") pod \"7fecb164-8181-44b0-9264-8d18ca7f1f60\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") "
Nov 01 13:00:20.510425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.510368    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-xks-ca\") pod \"7fecb164-8181-44b0-9264-8d18ca7f1f60\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") "
Nov 01 13:00:20.510425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.510406    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-openshift-ca\") pod \"7fecb164-8181-44b0-9264-8d18ca7f1f60\" (UID: \"7fecb164-8181-44b0-9264-8d18ca7f1f60\") "
Nov 01 13:00:20.510538 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.510485    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24d3559-6438-4932-b75a-6d5fb9825633-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24d3559-6438-4932-b75a-6d5fb9825633-maas-api-service-ca\") pod \"c24d3559-6438-4932-b75a-6d5fb9825633\" (UID: \"c24d3559-6438-4932-b75a-6d5fb9825633\") "
Nov 01 13:00:20.510639 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.510554    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24d3559-6438-4932-b75a-6d5fb9825633-kube-api-access-5gp2j\" (UniqueName: \"kubernetes.io/projected/c24d3559-6438-4932-b75a-6d5fb9825633-kube-api-access-5gp2j\") pod \"c24d3559-6438-4932-b75a-6d5fb9825633\" (UID: \"c24d3559-6438-4932-b75a-6d5fb9825633\") "
Nov 01 13:00:20.510848 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.510820    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-openshift-ca" pod "7fecb164-8181-44b0-9264-8d18ca7f1f60" (UID: "7fecb164-8181-44b0-9264-8d18ca7f1f60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:20.510990 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.510899    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24d3559-6438-4932-b75a-6d5fb9825633-maas-api-service-ca" pod "c24d3559-6438-4932-b75a-6d5fb9825633" (UID: "c24d3559-6438-4932-b75a-6d5fb9825633"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:20.510990 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.510916    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:20.512407 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.512386    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fecb164-8181-44b0-9264-8d18ca7f1f60-kube-api-access-t5thq" pod "7fecb164-8181-44b0-9264-8d18ca7f1f60" (UID: "7fecb164-8181-44b0-9264-8d18ca7f1f60"). InnerVolumeSpecName "kube-api-access-t5thq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:20.512518 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.512501    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-xks-ca" pod "7fecb164-8181-44b0-9264-8d18ca7f1f60" (UID: "7fecb164-8181-44b0-9264-8d18ca7f1f60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:20.512681 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.512660    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24d3559-6438-4932-b75a-6d5fb9825633-kube-api-access-5gp2j" pod "c24d3559-6438-4932-b75a-6d5fb9825633" (UID: "c24d3559-6438-4932-b75a-6d5fb9825633"). InnerVolumeSpecName "kube-api-access-5gp2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:20.612130 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.612092    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fecb164-8181-44b0-9264-8d18ca7f1f60-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:20.612130 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.612130    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24d3559-6438-4932-b75a-6d5fb9825633-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:20.612312 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.612145    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gp2j\" (UniqueName: \"kubernetes.io/projected/c24d3559-6438-4932-b75a-6d5fb9825633-kube-api-access-5gp2j\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:20.612312 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.612159    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5thq\" (UniqueName: \"kubernetes.io/projected/7fecb164-8181-44b0-9264-8d18ca7f1f60-kube-api-access-t5thq\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:20.618016 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.617983    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" path="/var/lib/kubelet/pods/79488639-741b-4eae-a32c-6315c2c6fe55/volumes"
Nov 01 13:00:20.618578 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:20.618541    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" path="/var/lib/kubelet/pods/a0309900-e26d-42bf-8138-cd5cdcc28d36/volumes"
Nov 01 13:00:21.244001 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:21.243966    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"
Nov 01 13:00:21.244509 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:21.243986    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6" event={"ID":"c24d3559-6438-4932-b75a-6d5fb9825633","Type":"ContainerDied","Data":"e0d5ea086c1870c6a0c3112205e6057d42f4fbce253e736e0296acb58cfbc8f5"}
Nov 01 13:00:21.244509 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:21.244087    5725 scope.go:122] "RemoveContainer" containerID="811a4df44ea7d8700719f668b06d25d5f2358b50702331a4217d3269bf8245ee"
Nov 01 13:00:21.246036 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:21.245800    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"
Nov 01 13:00:21.246036 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:21.245821    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl" event={"ID":"7fecb164-8181-44b0-9264-8d18ca7f1f60","Type":"ContainerDied","Data":"a8c584927098c854b83a0ce88fe729390bb15fd5cf6442f62b013e610eb2ac70"}
Nov 01 13:00:21.246036 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:21.245887    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8c584927098c854b83a0ce88fe729390bb15fd5cf6442f62b013e610eb2ac70"
Nov 01 13:00:21.460561 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:21.460520    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"]
Nov 01 13:00:21.460732 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:21.460580    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-72078a-29892300-lskk6"]
Nov 01 13:00:22.616816 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:22.616787    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" path="/var/lib/kubelet/pods/c24d3559-6438-4932-b75a-6d5fb9825633/volumes"
Nov 01 13:00:31.027425 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:31.027388    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:00:34.200921 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.200886    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:34.271678 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.271643    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6635c843-311e-4f2f-a17c-a797b86cd473-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6635c843-311e-4f2f-a17c-a797b86cd473-maas-api-service-ca\") pod \"6635c843-311e-4f2f-a17c-a797b86cd473\" (UID: \"6635c843-311e-4f2f-a17c-a797b86cd473\") "
Nov 01 13:00:34.271872 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.271823    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6635c843-311e-4f2f-a17c-a797b86cd473-kube-api-access-rj9q8\" (UniqueName: \"kubernetes.io/projected/6635c843-311e-4f2f-a17c-a797b86cd473-kube-api-access-rj9q8\") pod \"6635c843-311e-4f2f-a17c-a797b86cd473\" (UID: \"6635c843-311e-4f2f-a17c-a797b86cd473\") "
Nov 01 13:00:34.272036 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.272013    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6635c843-311e-4f2f-a17c-a797b86cd473-maas-api-service-ca" pod "6635c843-311e-4f2f-a17c-a797b86cd473" (UID: "6635c843-311e-4f2f-a17c-a797b86cd473"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:34.272278 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.272255    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6635c843-311e-4f2f-a17c-a797b86cd473-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:34.273934 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.273910    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6635c843-311e-4f2f-a17c-a797b86cd473-kube-api-access-rj9q8" pod "6635c843-311e-4f2f-a17c-a797b86cd473" (UID: "6635c843-311e-4f2f-a17c-a797b86cd473"). InnerVolumeSpecName "kube-api-access-rj9q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:34.345847 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.345809    5725 generic.go:356] "Generic (PLEG): container finished" podID="6635c843-311e-4f2f-a17c-a797b86cd473" containerID="5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042" exitCode=137
Nov 01 13:00:34.346166 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.346137    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb" event={"ID":"6635c843-311e-4f2f-a17c-a797b86cd473","Type":"ContainerDied","Data":"5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042"}
Nov 01 13:00:34.346379 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.346359    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb" event={"ID":"6635c843-311e-4f2f-a17c-a797b86cd473","Type":"ContainerDied","Data":"6eff772e2bbe13143fe4e3ddbbb4bf820ef3c77cbe167d109dc098feb9d2affb"}
Nov 01 13:00:34.346513 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.346498    5725 scope.go:122] "RemoveContainer" containerID="5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042"
Nov 01 13:00:34.346792 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.346771    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"
Nov 01 13:00:34.377379 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.377352    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj9q8\" (UniqueName: \"kubernetes.io/projected/6635c843-311e-4f2f-a17c-a797b86cd473-kube-api-access-rj9q8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:34.377549 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.377526    5725 scope.go:122] "RemoveContainer" containerID="f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a"
Nov 01 13:00:34.395470 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.395444    5725 scope.go:122] "RemoveContainer" containerID="5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042"
Nov 01 13:00:34.395817 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:34.395786    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042\": container with ID starting with 5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042 not found: ID does not exist" containerID="5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042"
Nov 01 13:00:34.395875 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.395828    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042"} err="failed to get container status \"5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042\": rpc error: code = NotFound desc = could not find container \"5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042\": container with ID starting with 5d886f1b9fb33e9bfbef31834133f33abb3d7c2087931cacab2f313a8c10c042 not found: ID does not exist"
Nov 01 13:00:34.395875 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.395854    5725 scope.go:122] "RemoveContainer" containerID="f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a"
Nov 01 13:00:34.396170 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:34.396126    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a\": container with ID starting with f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a not found: ID does not exist" containerID="f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a"
Nov 01 13:00:34.396241 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.396176    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a"} err="failed to get container status \"f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a\": rpc error: code = NotFound desc = could not find container \"f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a\": container with ID starting with f27dddbbfee7cbb74ddc5eebbdb37ce4dd6e9a66b07ae6c4b93e5ab1154da55a not found: ID does not exist"
Nov 01 13:00:34.506843 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.506811    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"]
Nov 01 13:00:34.509824 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.509797    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78c69d-29892300-vhrgb"]
Nov 01 13:00:34.617901 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:34.617824    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6635c843-311e-4f2f-a17c-a797b86cd473" path="/var/lib/kubelet/pods/6635c843-311e-4f2f-a17c-a797b86cd473/volumes"
Nov 01 13:00:42.372158 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:42.370846    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"]
Nov 01 13:00:42.372158 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:42.371132    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy" containerID="cri-o://3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff" gracePeriod=30
Nov 01 13:00:43.662027 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.661989    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"]
Nov 01 13:00:43.662564 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662550    5725 cpu_manager.go:414] "containerMap: removing container" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerName="cleanup"
Nov 01 13:00:43.662620 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662565    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerName="cleanup"
Nov 01 13:00:43.662620 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662579    5725 cpu_manager.go:414] "containerMap: removing container" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" containerName="cleanup"
Nov 01 13:00:43.662620 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662585    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" containerName="cleanup"
Nov 01 13:00:43.662620 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662596    5725 cpu_manager.go:414] "containerMap: removing container" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerName="cleanup"
Nov 01 13:00:43.662620 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662600    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerName="cleanup"
Nov 01 13:00:43.662620 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662608    5725 cpu_manager.go:414] "containerMap: removing container" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerName="cleanup"
Nov 01 13:00:43.662620 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662612    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerName="cleanup"
Nov 01 13:00:43.662620 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662620    5725 cpu_manager.go:414] "containerMap: removing container" podUID="6635c843-311e-4f2f-a17c-a797b86cd473" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662624    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6635c843-311e-4f2f-a17c-a797b86cd473" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662639    5725 cpu_manager.go:414] "containerMap: removing container" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662645    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662653    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8f69a0a6-7b30-47dc-b148-8e5700613790" containerName="revoke-keys"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662657    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f69a0a6-7b30-47dc-b148-8e5700613790" containerName="revoke-keys"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662664    5725 cpu_manager.go:414] "containerMap: removing container" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662668    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662675    5725 cpu_manager.go:414] "containerMap: removing container" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662679    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662693    5725 cpu_manager.go:414] "containerMap: removing container" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662698    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662704    5725 cpu_manager.go:414] "containerMap: removing container" podUID="7fecb164-8181-44b0-9264-8d18ca7f1f60" containerName="revoke-keys"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662708    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fecb164-8181-44b0-9264-8d18ca7f1f60" containerName="revoke-keys"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662778    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f69a0a6-7b30-47dc-b148-8e5700613790" containerName="revoke-keys"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662786    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662791    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662799    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662806    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fecb164-8181-44b0-9264-8d18ca7f1f60" containerName="revoke-keys"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662814    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662820    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="79488639-741b-4eae-a32c-6315c2c6fe55" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662826    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662831    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0309900-e26d-42bf-8138-cd5cdcc28d36" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662835    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="6635c843-311e-4f2f-a17c-a797b86cd473" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662843    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" containerName="cleanup"
Nov 01 13:00:43.662878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.662849    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="6635c843-311e-4f2f-a17c-a797b86cd473" containerName="cleanup"
Nov 01 13:00:43.667971 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.667953    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.808782 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.808748    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.808782 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.808796    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.809075 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.808901    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmzjk\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-kube-api-access-jmzjk\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.809075 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.809020    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.809075 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.809072    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.809263 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.809146    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.809263 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.809195    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.809376 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.809299    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01083685-36fb-4dd7-9d26-3619c1cdef31-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.809376 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.809340    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.855758 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.855712    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"]
Nov 01 13:00:43.910654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.910621    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.910654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.910659    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.910943 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.910688    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.910943 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.910727    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01083685-36fb-4dd7-9d26-3619c1cdef31-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.910943 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.910752    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.910943 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.910842    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.910943 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.910936    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.911260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.911104    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.911260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.911154    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmzjk\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-kube-api-access-jmzjk\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.911260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.911190    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.911260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.911154    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.911260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.911260    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.911523 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.911511    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.911584 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.911565    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01083685-36fb-4dd7-9d26-3619c1cdef31-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.913336 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.913281    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.913431 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.913373    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.965570 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.965523    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.965997 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.965975    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmzjk\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-kube-api-access-jmzjk\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:43.983918 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:43.983881    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:44.417277 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:44.417244    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"]
Nov 01 13:00:44.423240 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:00:44.423205    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01083685_36fb_4dd7_9d26_3619c1cdef31.slice/crio-0da4112042e225b42a87508716cfbd28158a4c06a1a2b22e73e9b74e9f0fefb1 WatchSource:0}: Error finding container 0da4112042e225b42a87508716cfbd28158a4c06a1a2b22e73e9b74e9f0fefb1: Status 404 returned error can't find the container with id 0da4112042e225b42a87508716cfbd28158a4c06a1a2b22e73e9b74e9f0fefb1
Nov 01 13:00:44.425429 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:44.425397    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:00:44.425515 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:44.425470    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:00:44.425515 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:44.425498    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:00:45.444075 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:45.441289    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" event={"ID":"01083685-36fb-4dd7-9d26-3619c1cdef31","Type":"ContainerStarted","Data":"1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287"}
Nov 01 13:00:45.444075 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:45.441359    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" event={"ID":"01083685-36fb-4dd7-9d26-3619c1cdef31","Type":"ContainerStarted","Data":"0da4112042e225b42a87508716cfbd28158a4c06a1a2b22e73e9b74e9f0fefb1"}
Nov 01 13:00:45.660929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:45.659723    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" podStartSLOduration=2.659699608 podStartE2EDuration="2.659699608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:45.602253011 +0000 UTC m=+2287.109444242" watchObservedRunningTime="2026-11-01 13:00:45.659699608 +0000 UTC m=+2287.166890830"
Nov 01 13:00:45.984273 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:45.984232    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:45.985596 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:45.985561    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 13:00:45.985722 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:45.985623    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 13:00:46.984259 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:46.984220    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 13:00:46.984773 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:46.984281    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 13:00:47.806031 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.805774    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 13:00:47.902527 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902488    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-envoy\") pod \"c61f02b5-3268-4b1a-b946-3c112ee12dec\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") "
Nov 01 13:00:47.902713 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902563    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-data\") pod \"c61f02b5-3268-4b1a-b946-3c112ee12dec\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") "
Nov 01 13:00:47.902713 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902606    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-credential-socket\") pod \"c61f02b5-3268-4b1a-b946-3c112ee12dec\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") "
Nov 01 13:00:47.902713 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902697    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-certs\") pod \"c61f02b5-3268-4b1a-b946-3c112ee12dec\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") "
Nov 01 13:00:47.902901 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902717    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-socket\") pod \"c61f02b5-3268-4b1a-b946-3c112ee12dec\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") "
Nov 01 13:00:47.902901 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902761    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-podinfo\") pod \"c61f02b5-3268-4b1a-b946-3c112ee12dec\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") "
Nov 01 13:00:47.902901 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902814    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-token\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-token\") pod \"c61f02b5-3268-4b1a-b946-3c112ee12dec\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") "
Nov 01 13:00:47.902901 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902854    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61f02b5-3268-4b1a-b946-3c112ee12dec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61f02b5-3268-4b1a-b946-3c112ee12dec-istiod-ca-cert\") pod \"c61f02b5-3268-4b1a-b946-3c112ee12dec\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") "
Nov 01 13:00:47.902901 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902893    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-kube-api-access-mdcrn\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-kube-api-access-mdcrn\") pod \"c61f02b5-3268-4b1a-b946-3c112ee12dec\" (UID: \"c61f02b5-3268-4b1a-b946-3c112ee12dec\") "
Nov 01 13:00:47.903211 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902909    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-data" pod "c61f02b5-3268-4b1a-b946-3c112ee12dec" (UID: "c61f02b5-3268-4b1a-b946-3c112ee12dec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:47.903211 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902967    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-socket" pod "c61f02b5-3268-4b1a-b946-3c112ee12dec" (UID: "c61f02b5-3268-4b1a-b946-3c112ee12dec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:47.903211 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.902995    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-credential-socket" pod "c61f02b5-3268-4b1a-b946-3c112ee12dec" (UID: "c61f02b5-3268-4b1a-b946-3c112ee12dec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:47.903391 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.903294    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61f02b5-3268-4b1a-b946-3c112ee12dec-istiod-ca-cert" pod "c61f02b5-3268-4b1a-b946-3c112ee12dec" (UID: "c61f02b5-3268-4b1a-b946-3c112ee12dec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:47.903391 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.903380    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:47.903510 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.903403    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:47.903510 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.903417    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:47.903510 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.903430    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61f02b5-3268-4b1a-b946-3c112ee12dec-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:47.903720 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.903700    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-certs" pod "c61f02b5-3268-4b1a-b946-3c112ee12dec" (UID: "c61f02b5-3268-4b1a-b946-3c112ee12dec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:47.905331 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.905292    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-envoy" pod "c61f02b5-3268-4b1a-b946-3c112ee12dec" (UID: "c61f02b5-3268-4b1a-b946-3c112ee12dec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:47.905910 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.905886    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-kube-api-access-mdcrn" pod "c61f02b5-3268-4b1a-b946-3c112ee12dec" (UID: "c61f02b5-3268-4b1a-b946-3c112ee12dec"). InnerVolumeSpecName "kube-api-access-mdcrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:47.906097 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.905949    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-token" pod "c61f02b5-3268-4b1a-b946-3c112ee12dec" (UID: "c61f02b5-3268-4b1a-b946-3c112ee12dec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:47.906206 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.905993    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-podinfo" pod "c61f02b5-3268-4b1a-b946-3c112ee12dec" (UID: "c61f02b5-3268-4b1a-b946-3c112ee12dec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:00:47.987600 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.987564    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 13:00:47.988223 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:47.988187    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 13:00:48.004544 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.004508    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdcrn\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-kube-api-access-mdcrn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.004686 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.004550    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.004686 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.004566    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61f02b5-3268-4b1a-b946-3c112ee12dec-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.004686 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.004579    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.004686 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.004593    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61f02b5-3268-4b1a-b946-3c112ee12dec-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.486640 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.486594    5725 generic.go:356] "Generic (PLEG): container finished" podID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerID="3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff" exitCode=0
Nov 01 13:00:48.486842 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.486642    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"
Nov 01 13:00:48.486842 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.486658    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" event={"ID":"c61f02b5-3268-4b1a-b946-3c112ee12dec","Type":"ContainerDied","Data":"3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff"}
Nov 01 13:00:48.486842 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.486717    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp" event={"ID":"c61f02b5-3268-4b1a-b946-3c112ee12dec","Type":"ContainerDied","Data":"fdc49fcd872a6da0ae604dc62b0679a07027a3e8a476509b45e603abe7f48d65"}
Nov 01 13:00:48.486842 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.486740    5725 scope.go:122] "RemoveContainer" containerID="3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff"
Nov 01 13:00:48.506612 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.506443    5725 scope.go:122] "RemoveContainer" containerID="3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff"
Nov 01 13:00:48.506834 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:48.506810    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff\": container with ID starting with 3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff not found: ID does not exist" containerID="3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff"
Nov 01 13:00:48.506891 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.506841    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff"} err="failed to get container status \"3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff\": rpc error: code = NotFound desc = could not find container \"3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff\": container with ID starting with 3ee89dc47084b3b33a729c26e85469b2068dd8b249e250a2625e52c4a65028ff not found: ID does not exist"
Nov 01 13:00:48.665241 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.665202    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"]
Nov 01 13:00:48.668393 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.668365    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-fmvsp"]
Nov 01 13:00:48.984786 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.984750    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 13:00:48.984964 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:48.984848    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 13:00:49.984419 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:49.984378    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 01 13:00:49.984803 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:49.984453    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 01 13:00:50.617757 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:50.617719    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" path="/var/lib/kubelet/pods/c61f02b5-3268-4b1a-b946-3c112ee12dec/volumes"
Nov 01 13:00:51.152125 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:51.152081    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:51.152917 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:51.152893    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:51.153669 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:51.153647    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:00:52.196307 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.196274    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"]
Nov 01 13:00:52.196793 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.196780    5725 cpu_manager.go:414] "containerMap: removing container" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" containerName="cleanup"
Nov 01 13:00:52.196855 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.196794    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" containerName="cleanup"
Nov 01 13:00:52.196855 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.196805    5725 cpu_manager.go:414] "containerMap: removing container" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy"
Nov 01 13:00:52.196855 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.196810    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy"
Nov 01 13:00:52.196855 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.196847    5725 cpu_manager.go:414] "containerMap: removing container" podUID="6635c843-311e-4f2f-a17c-a797b86cd473" containerName="cleanup"
Nov 01 13:00:52.196855 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.196852    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6635c843-311e-4f2f-a17c-a797b86cd473" containerName="cleanup"
Nov 01 13:00:52.197019 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.196932    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61f02b5-3268-4b1a-b946-3c112ee12dec" containerName="istio-proxy"
Nov 01 13:00:52.197019 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.196941    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24d3559-6438-4932-b75a-6d5fb9825633" containerName="cleanup"
Nov 01 13:00:52.201744 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.201723    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.253113 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.253083    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b68322-serving-cert\""
Nov 01 13:00:52.253874 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.253845    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"]
Nov 01 13:00:52.274849 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.274821    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.275061 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.274900    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2s4\" (UniqueName: \"kubernetes.io/projected/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-kube-api-access-9x2s4\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.275061 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.275010    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.275211 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.275085    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-metrics-certs\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.376594 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.376561    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.376758 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.376668    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2s4\" (UniqueName: \"kubernetes.io/projected/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-kube-api-access-9x2s4\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.376807 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.376771    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.376845 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.376831    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-metrics-certs\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.376925 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:52.376898    5725 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-serving-cert: secret "maas-api-e2e-inf-b-b68322-serving-cert" not found
Nov 01 13:00:52.377006 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:00:52.376988    5725 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls podName:e8b8f9f6-a438-4a81-9c2a-8de8db1a237d nodeName:}" failed. No retries permitted until 2026-11-01 13:00:52.876966953 +0000 UTC m=+2294.384158152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls") pod "maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k" (UID: "e8b8f9f6-a438-4a81-9c2a-8de8db1a237d") : secret "maas-api-e2e-inf-b-b68322-serving-cert" not found
Nov 01 13:00:52.377352 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.377332    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.379274 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.379253    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-metrics-certs\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.431176 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.431142    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2s4\" (UniqueName: \"kubernetes.io/projected/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-kube-api-access-9x2s4\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.884288 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.884245    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:52.886622 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:52.886600    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls\") pod \"maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:53.115898 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:53.115861    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:53.495555 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:53.495523    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"]
Nov 01 13:00:53.502586 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:00:53.502558    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b8f9f6_a438_4a81_9c2a_8de8db1a237d.slice/crio-3f6d4c56203b11b61b5eb23da3b62146bd6f17b11d3f55286af9098d2f3128cd WatchSource:0}: Error finding container 3f6d4c56203b11b61b5eb23da3b62146bd6f17b11d3f55286af9098d2f3128cd: Status 404 returned error can't find the container with id 3f6d4c56203b11b61b5eb23da3b62146bd6f17b11d3f55286af9098d2f3128cd
Nov 01 13:00:53.526832 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:53.526803    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k" event={"ID":"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d","Type":"ContainerStarted","Data":"3f6d4c56203b11b61b5eb23da3b62146bd6f17b11d3f55286af9098d2f3128cd"}
Nov 01 13:00:54.537241 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:54.537198    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k" event={"ID":"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d","Type":"ContainerStarted","Data":"c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1"}
Nov 01 13:00:54.537741 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:54.537283    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:00:54.660701 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:54.660645    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k" podStartSLOduration=1.726794323 podStartE2EDuration="2.660632543s" totalImagesPullingTime="933.83822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:54.659068675 +0000 UTC m=+2296.166259891" watchObservedRunningTime="2026-11-01 13:00:54.660632543 +0000 UTC m=+2296.167823760"
Nov 01 13:00:58.760968 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.760932    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"]
Nov 01 13:00:58.765234 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.765212    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.846196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.846158    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.846355 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.846222    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.846421 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.846349    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.846421 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.846406    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.846548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.846501    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gc5h\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-kube-api-access-8gc5h\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.846548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.846525    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.846548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.846541    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.846651 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.846560    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.846651 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.846597    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.869795 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.869756    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"]
Nov 01 13:00:58.947355 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947311    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.947355 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947353    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.947645 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947372    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.947645 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947507    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.947645 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947607    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.947835 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947705    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.947835 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947777    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.947835 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947815    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.947987 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947777    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.947987 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947949    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gc5h\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-kube-api-access-8gc5h\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.948139 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947998    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.948139 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.947992    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.948139 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.948110    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.948543 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.948525    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.949941 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.949917    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:58.950272 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:58.950254    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:59.001671 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.001633    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:59.001845 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.001675    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gc5h\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-kube-api-access-8gc5h\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:59.080756 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.080671    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:00:59.515674 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.515639    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"]
Nov 01 13:00:59.522075 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:00:59.522025    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4f885b_01d7_4675_9dff_d23d7f1499c1.slice/crio-7af7dd205306ef2f92ee301eb47584e60ea89171474af1e88ef3044e38cdebdc WatchSource:0}: Error finding container 7af7dd205306ef2f92ee301eb47584e60ea89171474af1e88ef3044e38cdebdc: Status 404 returned error can't find the container with id 7af7dd205306ef2f92ee301eb47584e60ea89171474af1e88ef3044e38cdebdc
Nov 01 13:00:59.524474 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.524438    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:00:59.524585 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.524521    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:00:59.524585 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.524566    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:00:59.571862 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.571838    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7091-ba45cd"]
Nov 01 13:00:59.577455 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.577430    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" event={"ID":"1e4f885b-01d7-4675-9dff-d23d7f1499c1","Type":"ContainerStarted","Data":"7af7dd205306ef2f92ee301eb47584e60ea89171474af1e88ef3044e38cdebdc"}
Nov 01 13:00:59.577554 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.577518    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:00:59.627619 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.627410    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-ba45cd"]
Nov 01 13:00:59.628175 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.628141    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rbvvl\""
Nov 01 13:00:59.657824 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.657778    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlhbl\" (UniqueName: \"kubernetes.io/projected/8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9-kube-api-access-xlhbl\") pod \"test-curl-7091-ba45cd\" (UID: \"8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9\") " pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:00:59.759403 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.759351    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlhbl\" (UniqueName: \"kubernetes.io/projected/8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9-kube-api-access-xlhbl\") pod \"test-curl-7091-ba45cd\" (UID: \"8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9\") " pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:00:59.814284 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.814198    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlhbl\" (UniqueName: \"kubernetes.io/projected/8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9-kube-api-access-xlhbl\") pod \"test-curl-7091-ba45cd\" (UID: \"8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9\") " pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:00:59.889848 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:00:59.889807    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:01:00.274263 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:00.274226    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-ba45cd"]
Nov 01 13:01:00.281459 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:01:00.281434    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b9efbd1_6f94_4c8c_93df_b0a1fb66dcc9.slice/crio-7cc0cfbf7624e613e6973f1a756de45308a00342d918a39a1d00349683ad1b56 WatchSource:0}: Error finding container 7cc0cfbf7624e613e6973f1a756de45308a00342d918a39a1d00349683ad1b56: Status 404 returned error can't find the container with id 7cc0cfbf7624e613e6973f1a756de45308a00342d918a39a1d00349683ad1b56
Nov 01 13:01:00.558797 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:00.558711    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:01:00.585424 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:00.585380    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-ba45cd" event={"ID":"8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9","Type":"ContainerStarted","Data":"7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb"}
Nov 01 13:01:00.585590 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:00.585438    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-ba45cd" event={"ID":"8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9","Type":"ContainerStarted","Data":"7cc0cfbf7624e613e6973f1a756de45308a00342d918a39a1d00349683ad1b56"}
Nov 01 13:01:00.587211 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:00.587184    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" event={"ID":"1e4f885b-01d7-4675-9dff-d23d7f1499c1","Type":"ContainerStarted","Data":"eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710"}
Nov 01 13:01:00.785481 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:00.785421    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" podStartSLOduration=2.785405983 podStartE2EDuration="2.785405983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:01:00.781990185 +0000 UTC m=+2302.289181413" watchObservedRunningTime="2026-11-01 13:01:00.785405983 +0000 UTC m=+2302.292597267"
Nov 01 13:01:00.905989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:00.905886    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7091-ba45cd" podStartSLOduration=1.905869652 podStartE2EDuration="1.905869652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:01:00.903184764 +0000 UTC m=+2302.410375982" watchObservedRunningTime="2026-11-01 13:01:00.905869652 +0000 UTC m=+2302.413060868"
Nov 01 13:01:01.049077 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:01.042241    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:01:01.081112 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:01.081078    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:01:01.082497 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:01.082473    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 01 13:01:01.082589 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:01.082525    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 01 13:01:02.081557 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:02.081523    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 01 13:01:02.081970 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:02.081577    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 01 13:01:02.356779 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:02.356192    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7091-ba45cd"]
Nov 01 13:01:02.361705 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:02.361536    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7091-ba45cd"]
Nov 01 13:01:02.602248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:02.602185    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7091-ba45cd" podUID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" containerName="test-curl-7091-ba45cd" containerID="cri-o://7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb" gracePeriod=2
Nov 01 13:01:02.652452 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:02.652330    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-ba45cd\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:01:03.081662 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:03.081626    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 01 13:01:03.082212 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:03.081687    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 01 13:01:03.658076 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:03.657449    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"]
Nov 01 13:01:03.658076 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:03.657712    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb" podUID="d957120b-c603-4caf-a992-b478e28008c5" containerName="maas-api" containerID="cri-o://bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea" gracePeriod=30
Nov 01 13:01:04.080904 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.080847    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 13:01:04.081395 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.081366    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 01 13:01:04.081540 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.081422    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 01 13:01:04.219494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.213350    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-metrics-certs\") pod \"d957120b-c603-4caf-a992-b478e28008c5\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") "
Nov 01 13:01:04.219494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.213895    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls\") pod \"d957120b-c603-4caf-a992-b478e28008c5\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") "
Nov 01 13:01:04.219494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.214336    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d957120b-c603-4caf-a992-b478e28008c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d957120b-c603-4caf-a992-b478e28008c5-openshift-service-ca\") pod \"d957120b-c603-4caf-a992-b478e28008c5\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") "
Nov 01 13:01:04.219494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.214377    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d957120b-c603-4caf-a992-b478e28008c5-kube-api-access-dqfs9\" (UniqueName: \"kubernetes.io/projected/d957120b-c603-4caf-a992-b478e28008c5-kube-api-access-dqfs9\") pod \"d957120b-c603-4caf-a992-b478e28008c5\" (UID: \"d957120b-c603-4caf-a992-b478e28008c5\") "
Nov 01 13:01:04.219494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.214770    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d957120b-c603-4caf-a992-b478e28008c5-openshift-service-ca" pod "d957120b-c603-4caf-a992-b478e28008c5" (UID: "d957120b-c603-4caf-a992-b478e28008c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:01:04.219494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.215008    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d957120b-c603-4caf-a992-b478e28008c5-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:04.219494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.218710    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-metrics-certs" pod "d957120b-c603-4caf-a992-b478e28008c5" (UID: "d957120b-c603-4caf-a992-b478e28008c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:04.219494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.218955    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls" pod "d957120b-c603-4caf-a992-b478e28008c5" (UID: "d957120b-c603-4caf-a992-b478e28008c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:04.222226 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.222197    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d957120b-c603-4caf-a992-b478e28008c5-kube-api-access-dqfs9" pod "d957120b-c603-4caf-a992-b478e28008c5" (UID: "d957120b-c603-4caf-a992-b478e28008c5"). InnerVolumeSpecName "kube-api-access-dqfs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:04.316593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.316513    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:04.316593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.316544    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d957120b-c603-4caf-a992-b478e28008c5-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:04.316593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.316554    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqfs9\" (UniqueName: \"kubernetes.io/projected/d957120b-c603-4caf-a992-b478e28008c5-kube-api-access-dqfs9\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:04.617395 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.617357    5725 generic.go:356] "Generic (PLEG): container finished" podID="d957120b-c603-4caf-a992-b478e28008c5" containerID="bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea" exitCode=0
Nov 01 13:01:04.617553 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.617411    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"
Nov 01 13:01:04.617553 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.617423    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb" event={"ID":"d957120b-c603-4caf-a992-b478e28008c5","Type":"ContainerDied","Data":"bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea"}
Nov 01 13:01:04.617553 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.617475    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb" event={"ID":"d957120b-c603-4caf-a992-b478e28008c5","Type":"ContainerDied","Data":"701a2f5a76237b453504651088ee5c17c3cf7ac4a66a900d68143db90b04baed"}
Nov 01 13:01:04.617553 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.617496    5725 scope.go:122] "RemoveContainer" containerID="bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea"
Nov 01 13:01:04.669482 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.669319    5725 scope.go:122] "RemoveContainer" containerID="bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea"
Nov 01 13:01:04.669731 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:04.669708    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea\": container with ID starting with bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea not found: ID does not exist" containerID="bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea"
Nov 01 13:01:04.669775 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.669740    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea"} err="failed to get container status \"bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea\": rpc error: code = NotFound desc = could not find container \"bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea\": container with ID starting with bbe3a38b7063fd39a97330f586cb2781867d07e6ec80bbb98494c010cc539dea not found: ID does not exist"
Nov 01 13:01:04.774879 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.774843    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"]
Nov 01 13:01:04.825072 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.823239    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2810c2-85cc59bc8d-z49cb"]
Nov 01 13:01:04.910953 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:04.910926    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:01:05.025199 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.025166    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9-kube-api-access-xlhbl\" (UniqueName: \"kubernetes.io/projected/8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9-kube-api-access-xlhbl\") pod \"8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9\" (UID: \"8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9\") "
Nov 01 13:01:05.037074 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.033199    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9-kube-api-access-xlhbl" pod "8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" (UID: "8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9"). InnerVolumeSpecName "kube-api-access-xlhbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:05.081618 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.081577    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body=
Nov 01 13:01:05.081824 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.081637    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused"
Nov 01 13:01:05.126521 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.126485    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlhbl\" (UniqueName: \"kubernetes.io/projected/8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9-kube-api-access-xlhbl\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:05.644903 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.644860    5725 generic.go:356] "Generic (PLEG): container finished" podID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" containerID="7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb" exitCode=137
Nov 01 13:01:05.645655 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.645014    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:01:05.645655 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.645306    5725 scope.go:122] "RemoveContainer" containerID="7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb"
Nov 01 13:01:05.664562 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.664535    5725 scope.go:122] "RemoveContainer" containerID="7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb"
Nov 01 13:01:05.665532 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:05.665034    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb\": container with ID starting with 7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb not found: ID does not exist" containerID="7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb"
Nov 01 13:01:05.665532 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:05.665112    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb"} err="failed to get container status \"7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb\": rpc error: code = NotFound desc = could not find container \"7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb\": container with ID starting with 7c724f48a5e33c98ab13bb555b8395d8c373d4b8d6a7ffeb288961f258af45bb not found: ID does not exist"
Nov 01 13:01:05.710355 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:05.710309    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-ba45cd\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:01:06.101149 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:06.101118    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:01:06.101973 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:06.101932    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:01:06.104032 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:06.104003    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:01:06.149652 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:06.149612    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-ba45cd\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:01:06.251965 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:06.251933    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"]
Nov 01 13:01:06.256568 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:06.256532    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2810c2-4bcmr"]
Nov 01 13:01:06.471242 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:06.471193    5725 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-ba45cd\" is forbidden: User \"system:node:ip-10-0-138-76.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-76.us-west-2.compute.internal' and this object" podUID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" pod="opendatahub/test-curl-7091-ba45cd"
Nov 01 13:01:06.617013 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:06.616972    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" path="/var/lib/kubelet/pods/8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9/volumes"
Nov 01 13:01:06.617487 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:06.617470    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f69a0a6-7b30-47dc-b148-8e5700613790" path="/var/lib/kubelet/pods/8f69a0a6-7b30-47dc-b148-8e5700613790/volumes"
Nov 01 13:01:06.617856 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:06.617842    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d957120b-c603-4caf-a992-b478e28008c5" path="/var/lib/kubelet/pods/d957120b-c603-4caf-a992-b478e28008c5/volumes"
Nov 01 13:01:18.286510 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:18.286474    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"]
Nov 01 13:01:18.333514 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:18.333480    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"]
Nov 01 13:01:18.336378 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:18.336345    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"]
Nov 01 13:01:23.931454 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:23.931414    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"]
Nov 01 13:01:23.932218 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:23.932195    5725 cpu_manager.go:414] "containerMap: removing container" podUID="d957120b-c603-4caf-a992-b478e28008c5" containerName="maas-api"
Nov 01 13:01:23.932218 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:23.932217    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d957120b-c603-4caf-a992-b478e28008c5" containerName="maas-api"
Nov 01 13:01:23.932392 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:23.932260    5725 cpu_manager.go:414] "containerMap: removing container" podUID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" containerName="test-curl-7091-ba45cd"
Nov 01 13:01:23.932392 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:23.932268    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" containerName="test-curl-7091-ba45cd"
Nov 01 13:01:23.932509 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:23.932428    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="d957120b-c603-4caf-a992-b478e28008c5" containerName="maas-api"
Nov 01 13:01:23.932509 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:23.932447    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9efbd1-6f94-4c8c-93df-b0a1fb66dcc9" containerName="test-curl-7091-ba45cd"
Nov 01 13:01:23.937610 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:23.937587    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.026795 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.026763    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"]
Nov 01 13:01:24.086892 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.086859    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.087082 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.086896    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.087082 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.086989    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.087082 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.087068    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.087197 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.087120    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.087197 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.087193    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.087307 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.087292    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.087345 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.087319    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.087392 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.087362    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcz2x\" (UniqueName: \"kubernetes.io/projected/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-kube-api-access-fcz2x\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189096 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.188977    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189096 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189018    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189096 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189041    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcz2x\" (UniqueName: \"kubernetes.io/projected/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-kube-api-access-fcz2x\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189096 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189083    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189475 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189198    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189475 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189255    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189475 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189298    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189475 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189317    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189475 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189421    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189843 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189495    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.189932 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.189898    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.190154 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.190134    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.190290 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.190271    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.190331 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.190276    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.191454 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.191433    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.191782 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.191766    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.243431 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.243403    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.243645 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.243618    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcz2x\" (UniqueName: \"kubernetes.io/projected/ff09df22-abb2-4797-a1c6-6eddaa1a60c0-kube-api-access-fcz2x\") pod \"maas-default-gateway-openshift-default-7df854bf67-ns4rd\" (UID: \"ff09df22-abb2-4797-a1c6-6eddaa1a60c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.256483 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.256455    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:24.691018 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.690981    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"]
Nov 01 13:01:24.698169 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:01:24.698143    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff09df22_abb2_4797_a1c6_6eddaa1a60c0.slice/crio-aa3afb6334093c7502af7be0c53bce4a4f7cbc59bfe671859296d395ee7ebfc0 WatchSource:0}: Error finding container aa3afb6334093c7502af7be0c53bce4a4f7cbc59bfe671859296d395ee7ebfc0: Status 404 returned error can't find the container with id aa3afb6334093c7502af7be0c53bce4a4f7cbc59bfe671859296d395ee7ebfc0
Nov 01 13:01:24.701411 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.701369    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:01:24.701520 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.701455    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:01:24.701520 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.701498    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:01:24.801670 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.801623    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" event={"ID":"ff09df22-abb2-4797-a1c6-6eddaa1a60c0","Type":"ContainerStarted","Data":"91798e0bc3fbdb428a202b703c800abac168b33b40d567ddca7848fcf32883fd"}
Nov 01 13:01:24.801845 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.801708    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" event={"ID":"ff09df22-abb2-4797-a1c6-6eddaa1a60c0","Type":"ContainerStarted","Data":"aa3afb6334093c7502af7be0c53bce4a4f7cbc59bfe671859296d395ee7ebfc0"}
Nov 01 13:01:24.913608 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:24.913556    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podStartSLOduration=1.913543558 podStartE2EDuration="1.913543558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:01:24.912321261 +0000 UTC m=+2326.419512478" watchObservedRunningTime="2026-11-01 13:01:24.913543558 +0000 UTC m=+2326.420734775"
Nov 01 13:01:25.258822 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:25.257470    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:25.259399 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:25.259323    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ns4rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 13:01:25.259399 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:25.259382    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podUID="ff09df22-abb2-4797-a1c6-6eddaa1a60c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 13:01:25.359729 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:25.359683    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"]
Nov 01 13:01:25.463149 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:25.463115    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"]
Nov 01 13:01:26.256856 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:26.256825    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ns4rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 13:01:26.257083 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:26.256918    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podUID="ff09df22-abb2-4797-a1c6-6eddaa1a60c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 13:01:27.258430 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:27.258397    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ns4rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 13:01:27.258940 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:27.258448    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podUID="ff09df22-abb2-4797-a1c6-6eddaa1a60c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 13:01:28.258573 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:28.258524    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ns4rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 13:01:28.258996 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:28.258592    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podUID="ff09df22-abb2-4797-a1c6-6eddaa1a60c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 13:01:29.257584 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:29.257550    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ns4rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 13:01:29.257778 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:29.257605    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podUID="ff09df22-abb2-4797-a1c6-6eddaa1a60c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 13:01:29.596832 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:29.596726    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"]
Nov 01 13:01:29.597954 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:29.597883    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9" podUID="eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" containerName="maas-api" containerID="cri-o://7ff11ec02712baf2e045154af4b6028fd56e3a09b93b50cf7e04d69bdfb0154b" gracePeriod=30
Nov 01 13:01:29.900019 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:29.899918    5725 generic.go:356] "Generic (PLEG): container finished" podID="eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" containerID="7ff11ec02712baf2e045154af4b6028fd56e3a09b93b50cf7e04d69bdfb0154b" exitCode=0
Nov 01 13:01:29.900233 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:29.900080    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9" event={"ID":"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1","Type":"ContainerDied","Data":"7ff11ec02712baf2e045154af4b6028fd56e3a09b93b50cf7e04d69bdfb0154b"}
Nov 01 13:01:30.005951 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.005919    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 13:01:30.089227 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.089191    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls\") pod \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") "
Nov 01 13:01:30.089558 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.089537    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-kube-api-access-h2v5r\" (UniqueName: \"kubernetes.io/projected/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-kube-api-access-h2v5r\") pod \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") "
Nov 01 13:01:30.089821 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.089803    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-metrics-certs\") pod \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") "
Nov 01 13:01:30.089983 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.089965    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-openshift-service-ca\") pod \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\" (UID: \"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1\") "
Nov 01 13:01:30.090646 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.090360    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-openshift-service-ca" pod "eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" (UID: "eaa6b7da-b631-4bd0-9487-073cf7ab1ec1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:01:30.090646 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.090616    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:30.092102 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.092081    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls" pod "eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" (UID: "eaa6b7da-b631-4bd0-9487-073cf7ab1ec1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:30.092209 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.092194    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-kube-api-access-h2v5r" pod "eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" (UID: "eaa6b7da-b631-4bd0-9487-073cf7ab1ec1"). InnerVolumeSpecName "kube-api-access-h2v5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:30.092281 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.092255    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-metrics-certs" pod "eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" (UID: "eaa6b7da-b631-4bd0-9487-073cf7ab1ec1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:30.192229 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.192151    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:30.192229 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.192182    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2v5r\" (UniqueName: \"kubernetes.io/projected/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-kube-api-access-h2v5r\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:30.192229 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.192193    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:30.259076 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.258497    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ns4rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 13:01:30.259076 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.258556    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podUID="ff09df22-abb2-4797-a1c6-6eddaa1a60c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 13:01:30.360613 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.360571    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6" podUID="40cb0bff-44e1-409a-b7fe-891015b30fcb" containerName="payload-processing" containerID="cri-o://729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e" gracePeriod=25
Nov 01 13:01:30.464487 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.464397    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5" podUID="a0c585c6-9d66-4a3c-b13e-c6004c079867" containerName="payload-pre-processing" containerID="cri-o://9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4" gracePeriod=25
Nov 01 13:01:30.909242 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.909205    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9" event={"ID":"eaa6b7da-b631-4bd0-9487-073cf7ab1ec1","Type":"ContainerDied","Data":"c992638911374a6549c4ff15f824408c7d3b7820f74ac948753736acfa5a9555"}
Nov 01 13:01:30.909732 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.909246    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"
Nov 01 13:01:30.909732 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:30.909270    5725 scope.go:122] "RemoveContainer" containerID="7ff11ec02712baf2e045154af4b6028fd56e3a09b93b50cf7e04d69bdfb0154b"
Nov 01 13:01:31.035590 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:31.035538    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:01:31.065583 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:31.065549    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"]
Nov 01 13:01:31.112320 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:31.112281    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-187d06ed-66f5cdd9ff-vlrl9"]
Nov 01 13:01:31.257713 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:31.257671    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ns4rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 13:01:31.257940 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:31.257734    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podUID="ff09df22-abb2-4797-a1c6-6eddaa1a60c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 13:01:32.257723 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:32.257681    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ns4rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 13:01:32.258218 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:32.257767    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podUID="ff09df22-abb2-4797-a1c6-6eddaa1a60c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 13:01:32.618708 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:32.618615    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" path="/var/lib/kubelet/pods/eaa6b7da-b631-4bd0-9487-073cf7ab1ec1/volumes"
Nov 01 13:01:33.312291 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.312243    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" containerID="cri-o://aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7" gracePeriod=45
Nov 01 13:01:33.354510 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.354467    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main" containerID="cri-o://6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e" gracePeriod=45
Nov 01 13:01:33.356982 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.356414    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main" containerID="cri-o://1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6" gracePeriod=45
Nov 01 13:01:33.379960 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.379909    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:33.380409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.380364    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:33.384465 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.384437    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd"
Nov 01 13:01:33.437480 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.437443    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": dial tcp 10.132.0.107:8000: connect: connection refused"
Nov 01 13:01:33.474141 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.474109    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5" podUID="a0c585c6-9d66-4a3c-b13e-c6004c079867" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:01:33.474535 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.474502    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6" podUID="40cb0bff-44e1-409a-b7fe-891015b30fcb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:01:33.704072 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.699433    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 13:01:33.746182 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.746108    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 13:01:33.751202 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.751169    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 13:01:33.836463 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836360    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-home\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-home\") pod \"0decb764-723c-4026-95d9-8096aabf3571\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") "
Nov 01 13:01:33.836463 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836450    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-model-cache\") pod \"85276343-99cd-427a-b0b2-2eea7e0951bf\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") "
Nov 01 13:01:33.836711 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836486    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-tmp-dir\") pod \"0decb764-723c-4026-95d9-8096aabf3571\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") "
Nov 01 13:01:33.836711 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836516    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-model-cache\") pod \"0decb764-723c-4026-95d9-8096aabf3571\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") "
Nov 01 13:01:33.836711 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836551    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0decb764-723c-4026-95d9-8096aabf3571-tls-certs\" (UniqueName: \"kubernetes.io/secret/0decb764-723c-4026-95d9-8096aabf3571-tls-certs\") pod \"0decb764-723c-4026-95d9-8096aabf3571\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") "
Nov 01 13:01:33.836711 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836593    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-home\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-home\") pod \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") "
Nov 01 13:01:33.836711 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836625    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-tmp-dir\") pod \"85276343-99cd-427a-b0b2-2eea7e0951bf\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") "
Nov 01 13:01:33.836711 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836667    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tls-certs\") pod \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") "
Nov 01 13:01:33.836711 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836696    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85276343-99cd-427a-b0b2-2eea7e0951bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/85276343-99cd-427a-b0b2-2eea7e0951bf-tls-certs\") pod \"85276343-99cd-427a-b0b2-2eea7e0951bf\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836733    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-dshm\") pod \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836766    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-model-cache\") pod \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836810    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-model-cache" pod "0decb764-723c-4026-95d9-8096aabf3571" (UID: "0decb764-723c-4026-95d9-8096aabf3571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836844    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-dshm\") pod \"0decb764-723c-4026-95d9-8096aabf3571\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836882    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0decb764-723c-4026-95d9-8096aabf3571-kube-api-access-92zp4\" (UniqueName: \"kubernetes.io/projected/0decb764-723c-4026-95d9-8096aabf3571-kube-api-access-92zp4\") pod \"0decb764-723c-4026-95d9-8096aabf3571\" (UID: \"0decb764-723c-4026-95d9-8096aabf3571\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836920    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85276343-99cd-427a-b0b2-2eea7e0951bf-kube-api-access-cs2nz\" (UniqueName: \"kubernetes.io/projected/85276343-99cd-427a-b0b2-2eea7e0951bf-kube-api-access-cs2nz\") pod \"85276343-99cd-427a-b0b2-2eea7e0951bf\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836959    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-kube-api-access-qcjqc\" (UniqueName: \"kubernetes.io/projected/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-kube-api-access-qcjqc\") pod \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.836995    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-home\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-home\") pod \"85276343-99cd-427a-b0b2-2eea7e0951bf\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.837034    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-dshm\") pod \"85276343-99cd-427a-b0b2-2eea7e0951bf\" (UID: \"85276343-99cd-427a-b0b2-2eea7e0951bf\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.837075    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tmp-dir\") pod \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\" (UID: \"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d\") "
Nov 01 13:01:33.837248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.837150    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-tmp-dir" pod "0decb764-723c-4026-95d9-8096aabf3571" (UID: "0decb764-723c-4026-95d9-8096aabf3571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.837907 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.837329    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tmp-dir" pod "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" (UID: "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.837907 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.837563    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-home" pod "0decb764-723c-4026-95d9-8096aabf3571" (UID: "0decb764-723c-4026-95d9-8096aabf3571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.837907 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.837705    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.837907 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.837786    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.837907 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.837802    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.837907 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.837817    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.838264 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.838029    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-home" pod "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" (UID: "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.838264 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.838039    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-model-cache" pod "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" (UID: "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.838530 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.838504    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-tmp-dir" pod "85276343-99cd-427a-b0b2-2eea7e0951bf" (UID: "85276343-99cd-427a-b0b2-2eea7e0951bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.838805 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.838781    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-model-cache" pod "85276343-99cd-427a-b0b2-2eea7e0951bf" (UID: "85276343-99cd-427a-b0b2-2eea7e0951bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.839096 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.839076    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-home" pod "85276343-99cd-427a-b0b2-2eea7e0951bf" (UID: "85276343-99cd-427a-b0b2-2eea7e0951bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.840212 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.840186    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85276343-99cd-427a-b0b2-2eea7e0951bf-tls-certs" pod "85276343-99cd-427a-b0b2-2eea7e0951bf" (UID: "85276343-99cd-427a-b0b2-2eea7e0951bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:33.841531 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.841484    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-dshm" pod "85276343-99cd-427a-b0b2-2eea7e0951bf" (UID: "85276343-99cd-427a-b0b2-2eea7e0951bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.843027 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.843001    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85276343-99cd-427a-b0b2-2eea7e0951bf-kube-api-access-cs2nz" pod "85276343-99cd-427a-b0b2-2eea7e0951bf" (UID: "85276343-99cd-427a-b0b2-2eea7e0951bf"). InnerVolumeSpecName "kube-api-access-cs2nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:33.844449 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.844423    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0decb764-723c-4026-95d9-8096aabf3571-tls-certs" pod "0decb764-723c-4026-95d9-8096aabf3571" (UID: "0decb764-723c-4026-95d9-8096aabf3571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:33.844574 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.844430    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-dshm" pod "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" (UID: "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.844774 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.844723    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-dshm" pod "0decb764-723c-4026-95d9-8096aabf3571" (UID: "0decb764-723c-4026-95d9-8096aabf3571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:33.844854 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.844832    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-kube-api-access-qcjqc" pod "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" (UID: "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d"). InnerVolumeSpecName "kube-api-access-qcjqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:33.846652 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.846624    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0decb764-723c-4026-95d9-8096aabf3571-kube-api-access-92zp4" pod "0decb764-723c-4026-95d9-8096aabf3571" (UID: "0decb764-723c-4026-95d9-8096aabf3571"). InnerVolumeSpecName "kube-api-access-92zp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:33.849297 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.849267    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tls-certs" pod "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" (UID: "e3eec3d7-74ed-42cf-9b7f-197ba51daf8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:33.941853 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.941819    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942116 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942100    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942222 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942211    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942308 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942298    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85276343-99cd-427a-b0b2-2eea7e0951bf-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942388 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942379    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942475 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942463    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942559 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942549    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0decb764-723c-4026-95d9-8096aabf3571-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942644 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942634    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92zp4\" (UniqueName: \"kubernetes.io/projected/0decb764-723c-4026-95d9-8096aabf3571-kube-api-access-92zp4\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942722 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942711    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs2nz\" (UniqueName: \"kubernetes.io/projected/85276343-99cd-427a-b0b2-2eea7e0951bf-kube-api-access-cs2nz\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942805 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942794    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcjqc\" (UniqueName: \"kubernetes.io/projected/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d-kube-api-access-qcjqc\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942891 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942881    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.942966 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.942957    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.943061 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.943035    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85276343-99cd-427a-b0b2-2eea7e0951bf-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.943156 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.943144    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0decb764-723c-4026-95d9-8096aabf3571-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.949537 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.949503    5725 generic.go:356] "Generic (PLEG): container finished" podID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerID="1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6" exitCode=0
Nov 01 13:01:33.955067 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.949776    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"
Nov 01 13:01:33.955067 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.949796    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" event={"ID":"85276343-99cd-427a-b0b2-2eea7e0951bf","Type":"ContainerDied","Data":"1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6"}
Nov 01 13:01:33.955067 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.950026    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4" event={"ID":"85276343-99cd-427a-b0b2-2eea7e0951bf","Type":"ContainerDied","Data":"cacecdb09033ad771df761e82c4cf4369abca3cfcefe0672960515c216980c33"}
Nov 01 13:01:33.955067 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.950104    5725 scope.go:122] "RemoveContainer" containerID="1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6"
Nov 01 13:01:33.957540 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.957514    5725 generic.go:356] "Generic (PLEG): container finished" podID="0decb764-723c-4026-95d9-8096aabf3571" containerID="6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e" exitCode=0
Nov 01 13:01:33.957669 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.957549    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"
Nov 01 13:01:33.957669 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.957599    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" event={"ID":"0decb764-723c-4026-95d9-8096aabf3571","Type":"ContainerDied","Data":"6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e"}
Nov 01 13:01:33.957669 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.957630    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf" event={"ID":"0decb764-723c-4026-95d9-8096aabf3571","Type":"ContainerDied","Data":"3c29c77408f64cdef45bb799e547100db6035787767eb11ea137851b85d325a3"}
Nov 01 13:01:33.959349 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.959208    5725 generic.go:356] "Generic (PLEG): container finished" podID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerID="aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7" exitCode=0
Nov 01 13:01:33.959349 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.959257    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"
Nov 01 13:01:33.959349 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.959319    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" event={"ID":"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d","Type":"ContainerDied","Data":"aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7"}
Nov 01 13:01:33.959567 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.959356    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh" event={"ID":"e3eec3d7-74ed-42cf-9b7f-197ba51daf8d","Type":"ContainerDied","Data":"f2cfb583388905c3bfef83a58faed403aadf47f70bf46aa62a0e0e4499d6ee9c"}
Nov 01 13:01:33.977362 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.977339    5725 scope.go:122] "RemoveContainer" containerID="1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6"
Nov 01 13:01:33.977736 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:33.977712    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6\": container with ID starting with 1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6 not found: ID does not exist" containerID="1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6"
Nov 01 13:01:33.977819 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.977749    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6"} err="failed to get container status \"1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6\": rpc error: code = NotFound desc = could not find container \"1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6\": container with ID starting with 1ddbdc905e2907fdc3a6425d4bdaac4404ee7d7cd743f060a54000a6f593efd6 not found: ID does not exist"
Nov 01 13:01:33.977819 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.977775    5725 scope.go:122] "RemoveContainer" containerID="6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e"
Nov 01 13:01:33.990572 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.990549    5725 scope.go:122] "RemoveContainer" containerID="6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e"
Nov 01 13:01:33.990889 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:33.990870    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e\": container with ID starting with 6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e not found: ID does not exist" containerID="6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e"
Nov 01 13:01:33.990985 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.990904    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e"} err="failed to get container status \"6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e\": rpc error: code = NotFound desc = could not find container \"6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e\": container with ID starting with 6eed0504bf605bc6cd48533388536166e3d23b38c4c943608f24a0f5361eef0e not found: ID does not exist"
Nov 01 13:01:33.990985 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:33.990929    5725 scope.go:122] "RemoveContainer" containerID="aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7"
Nov 01 13:01:34.000891 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.000868    5725 scope.go:122] "RemoveContainer" containerID="aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7"
Nov 01 13:01:34.001208 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:34.001183    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7\": container with ID starting with aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7 not found: ID does not exist" containerID="aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7"
Nov 01 13:01:34.001296 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.001218    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7"} err="failed to get container status \"aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7\": rpc error: code = NotFound desc = could not find container \"aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7\": container with ID starting with aef266af10ae5eb27af4e2cb0d5443b4c799c9373364832f1cd15a0ba80220c7 not found: ID does not exist"
Nov 01 13:01:34.161458 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.161348    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"]
Nov 01 13:01:34.171154 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.171116    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/distinct-w5-9514b0-kserve-789b75cb8f-j4xrh"]
Nov 01 13:01:34.271351 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.271311    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"]
Nov 01 13:01:34.274175 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.274146    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/premium-w5-9514b0-kserve-5cf5658fbb-rjxkf"]
Nov 01 13:01:34.376964 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.376889    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"]
Nov 01 13:01:34.380130 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.380104    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9514b0/simulator-w5-9514b0-kserve-65d6557df5-ml6d4"]
Nov 01 13:01:34.617818 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.617784    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0decb764-723c-4026-95d9-8096aabf3571" path="/var/lib/kubelet/pods/0decb764-723c-4026-95d9-8096aabf3571/volumes"
Nov 01 13:01:34.618301 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.618286    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" path="/var/lib/kubelet/pods/85276343-99cd-427a-b0b2-2eea7e0951bf/volumes"
Nov 01 13:01:34.618708 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:34.618695    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" path="/var/lib/kubelet/pods/e3eec3d7-74ed-42cf-9b7f-197ba51daf8d/volumes"
Nov 01 13:01:37.358902 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:37.358864    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"]
Nov 01 13:01:43.473392 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:43.473352    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5" podUID="a0c585c6-9d66-4a3c-b13e-c6004c079867" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:01:43.473913 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:43.473410    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6" podUID="40cb0bff-44e1-409a-b7fe-891015b30fcb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:01:44.394609 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.394571    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"]
Nov 01 13:01:44.395260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395236    5725 cpu_manager.go:414] "containerMap: removing container" podUID="eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" containerName="maas-api"
Nov 01 13:01:44.395260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395256    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" containerName="maas-api"
Nov 01 13:01:44.395425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395274    5725 cpu_manager.go:414] "containerMap: removing container" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main"
Nov 01 13:01:44.395425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395279    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main"
Nov 01 13:01:44.395425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395305    5725 cpu_manager.go:414] "containerMap: removing container" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main"
Nov 01 13:01:44.395425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395310    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main"
Nov 01 13:01:44.395425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395324    5725 cpu_manager.go:414] "containerMap: removing container" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main"
Nov 01 13:01:44.395425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395329    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main"
Nov 01 13:01:44.395425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395406    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="85276343-99cd-427a-b0b2-2eea7e0951bf" containerName="main"
Nov 01 13:01:44.395425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395415    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="0decb764-723c-4026-95d9-8096aabf3571" containerName="main"
Nov 01 13:01:44.395425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395425    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa6b7da-b631-4bd0-9487-073cf7ab1ec1" containerName="maas-api"
Nov 01 13:01:44.395779 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.395432    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3eec3d7-74ed-42cf-9b7f-197ba51daf8d" containerName="main"
Nov 01 13:01:44.400870 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.400848    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.461593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.460733    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-model-cache\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.461593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.460791    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-tmp-dir\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.461593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.460817    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-dshm\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.461593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.460917    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-home\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.461593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.460960    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5smh\" (UniqueName: \"kubernetes.io/projected/d25850a7-529b-41db-8318-054a24043526-kube-api-access-z5smh\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.461593 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.461034    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25850a7-529b-41db-8318-054a24043526-tls-certs\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.487586 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.487553    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"]
Nov 01 13:01:44.506496 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.506453    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72078a\"/\"openshift-service-ca.crt\""
Nov 01 13:01:44.506496 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.506457    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72078a\"/\"kube-root-ca.crt\""
Nov 01 13:01:44.507086 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.507063    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72078a\"/\"default-dockercfg-lqhwg\""
Nov 01 13:01:44.507218 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.507123    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72078a\"/\"test-model-72078a-kserve-self-signed-certs\""
Nov 01 13:01:44.562236 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.562203    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-home\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.562417 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.562255    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5smh\" (UniqueName: \"kubernetes.io/projected/d25850a7-529b-41db-8318-054a24043526-kube-api-access-z5smh\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.562417 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.562310    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25850a7-529b-41db-8318-054a24043526-tls-certs\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.562417 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.562354    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-model-cache\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.562417 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.562376    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-tmp-dir\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.562417 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.562391    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-dshm\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.562723 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.562698    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-home\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.562783 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.562725    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-model-cache\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.562833 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.562793    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-tmp-dir\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.565066 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.565031    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-dshm\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.565498 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.565458    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25850a7-529b-41db-8318-054a24043526-tls-certs\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.637094 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.628594    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5smh\" (UniqueName: \"kubernetes.io/projected/d25850a7-529b-41db-8318-054a24043526-kube-api-access-z5smh\") pod \"test-model-72078a-kserve-8696584c7c-jkghk\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") " pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:44.712985 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:44.712892    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:45.107213 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:45.107176    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"]
Nov 01 13:01:45.113610 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:01:45.113575    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd25850a7_529b_41db_8318_054a24043526.slice/crio-e8d1dccbc2d346390377a0b1a5bfee8d99ed8cd3776e43d55cbea40924f1ea79 WatchSource:0}: Error finding container e8d1dccbc2d346390377a0b1a5bfee8d99ed8cd3776e43d55cbea40924f1ea79: Status 404 returned error can't find the container with id e8d1dccbc2d346390377a0b1a5bfee8d99ed8cd3776e43d55cbea40924f1ea79
Nov 01 13:01:46.048778 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:46.048742    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk" event={"ID":"d25850a7-529b-41db-8318-054a24043526","Type":"ContainerStarted","Data":"b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167"}
Nov 01 13:01:46.049243 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:46.048789    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk" event={"ID":"d25850a7-529b-41db-8318-054a24043526","Type":"ContainerStarted","Data":"e8d1dccbc2d346390377a0b1a5bfee8d99ed8cd3776e43d55cbea40924f1ea79"}
Nov 01 13:01:46.160748 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:46.160696    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk" podStartSLOduration=2.160682838 podStartE2EDuration="2.160682838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:01:46.158197012 +0000 UTC m=+2347.665388228" watchObservedRunningTime="2026-11-01 13:01:46.160682838 +0000 UTC m=+2347.667874054"
Nov 01 13:01:46.713749 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:46.713717    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:50.664721 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.663575    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 13:01:50.740482 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.740450    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 13:01:50.742681 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.742661    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40cb0bff-44e1-409a-b7fe-891015b30fcb-kube-api-access-wcw9j\" (UniqueName: \"kubernetes.io/projected/40cb0bff-44e1-409a-b7fe-891015b30fcb-kube-api-access-wcw9j\") pod \"40cb0bff-44e1-409a-b7fe-891015b30fcb\" (UID: \"40cb0bff-44e1-409a-b7fe-891015b30fcb\") "
Nov 01 13:01:50.742754 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.742712    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40cb0bff-44e1-409a-b7fe-891015b30fcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cb0bff-44e1-409a-b7fe-891015b30fcb-plugins-config-volume\") pod \"40cb0bff-44e1-409a-b7fe-891015b30fcb\" (UID: \"40cb0bff-44e1-409a-b7fe-891015b30fcb\") "
Nov 01 13:01:50.743190 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.743165    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cb0bff-44e1-409a-b7fe-891015b30fcb-plugins-config-volume" pod "40cb0bff-44e1-409a-b7fe-891015b30fcb" (UID: "40cb0bff-44e1-409a-b7fe-891015b30fcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:01:50.744802 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.744781    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40cb0bff-44e1-409a-b7fe-891015b30fcb-kube-api-access-wcw9j" pod "40cb0bff-44e1-409a-b7fe-891015b30fcb" (UID: "40cb0bff-44e1-409a-b7fe-891015b30fcb"). InnerVolumeSpecName "kube-api-access-wcw9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:50.844335 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.844298    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c585c6-9d66-4a3c-b13e-c6004c079867-kube-api-access-6sh49\" (UniqueName: \"kubernetes.io/projected/a0c585c6-9d66-4a3c-b13e-c6004c079867-kube-api-access-6sh49\") pod \"a0c585c6-9d66-4a3c-b13e-c6004c079867\" (UID: \"a0c585c6-9d66-4a3c-b13e-c6004c079867\") "
Nov 01 13:01:50.844519 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.844470    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c585c6-9d66-4a3c-b13e-c6004c079867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c585c6-9d66-4a3c-b13e-c6004c079867-plugins-config-volume\") pod \"a0c585c6-9d66-4a3c-b13e-c6004c079867\" (UID: \"a0c585c6-9d66-4a3c-b13e-c6004c079867\") "
Nov 01 13:01:50.844824 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.844802    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcw9j\" (UniqueName: \"kubernetes.io/projected/40cb0bff-44e1-409a-b7fe-891015b30fcb-kube-api-access-wcw9j\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:50.844899 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.844825    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cb0bff-44e1-409a-b7fe-891015b30fcb-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:50.844899 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.844843    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c585c6-9d66-4a3c-b13e-c6004c079867-plugins-config-volume" pod "a0c585c6-9d66-4a3c-b13e-c6004c079867" (UID: "a0c585c6-9d66-4a3c-b13e-c6004c079867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:01:50.846562 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.846534    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c585c6-9d66-4a3c-b13e-c6004c079867-kube-api-access-6sh49" pod "a0c585c6-9d66-4a3c-b13e-c6004c079867" (UID: "a0c585c6-9d66-4a3c-b13e-c6004c079867"). InnerVolumeSpecName "kube-api-access-6sh49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:50.946463 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.946368    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sh49\" (UniqueName: \"kubernetes.io/projected/a0c585c6-9d66-4a3c-b13e-c6004c079867-kube-api-access-6sh49\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:50.946463 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:50.946407    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c585c6-9d66-4a3c-b13e-c6004c079867-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:51.088029 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.087997    5725 generic.go:356] "Generic (PLEG): container finished" podID="a0c585c6-9d66-4a3c-b13e-c6004c079867" containerID="9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4" exitCode=0
Nov 01 13:01:51.088260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.088064    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"
Nov 01 13:01:51.088260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.088078    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5" event={"ID":"a0c585c6-9d66-4a3c-b13e-c6004c079867","Type":"ContainerDied","Data":"9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4"}
Nov 01 13:01:51.088260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.088132    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5" event={"ID":"a0c585c6-9d66-4a3c-b13e-c6004c079867","Type":"ContainerDied","Data":"2696e552b40d792a5671e038c703e8680c414bfd3fd5d00d173558425d3b8ba1"}
Nov 01 13:01:51.088260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.088165    5725 scope.go:122] "RemoveContainer" containerID="9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4"
Nov 01 13:01:51.089810 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.089780    5725 generic.go:356] "Generic (PLEG): container finished" podID="40cb0bff-44e1-409a-b7fe-891015b30fcb" containerID="729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e" exitCode=0
Nov 01 13:01:51.089929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.089831    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"
Nov 01 13:01:51.089929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.089832    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6" event={"ID":"40cb0bff-44e1-409a-b7fe-891015b30fcb","Type":"ContainerDied","Data":"729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e"}
Nov 01 13:01:51.089929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.089869    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6" event={"ID":"40cb0bff-44e1-409a-b7fe-891015b30fcb","Type":"ContainerDied","Data":"a24b8b024066bcfdc1a26137cfe00c0a36cc20acd9ce83911d02ea96399e54c4"}
Nov 01 13:01:51.099710 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.099688    5725 scope.go:122] "RemoveContainer" containerID="9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4"
Nov 01 13:01:51.100083 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:51.100029    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4\": container with ID starting with 9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4 not found: ID does not exist" containerID="9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4"
Nov 01 13:01:51.100178 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.100073    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4"} err="failed to get container status \"9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4\": rpc error: code = NotFound desc = could not find container \"9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4\": container with ID starting with 9df79619271335f4b6ebc53c4ada6da3e3694e253c4412fed3e2e20e33d468c4 not found: ID does not exist"
Nov 01 13:01:51.100178 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.100097    5725 scope.go:122] "RemoveContainer" containerID="729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e"
Nov 01 13:01:51.109616 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.109595    5725 scope.go:122] "RemoveContainer" containerID="729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e"
Nov 01 13:01:51.109919 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:51.109900    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e\": container with ID starting with 729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e not found: ID does not exist" containerID="729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e"
Nov 01 13:01:51.109973 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.109926    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e"} err="failed to get container status \"729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e\": rpc error: code = NotFound desc = could not find container \"729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e\": container with ID starting with 729acc6b1571363d116c683db07a9be8e352ccfebece993bda56dcd9c8b2235e not found: ID does not exist"
Nov 01 13:01:51.243709 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.243673    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"]
Nov 01 13:01:51.248970 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.248935    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9514b0-644557dd87-94xr6"]
Nov 01 13:01:51.349648 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.349610    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"]
Nov 01 13:01:51.353873 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:51.353842    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9514b0-66cdb47467-lxvf5"]
Nov 01 13:01:52.381501 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.381459    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c" podUID="1757ffa6-b479-45f0-bb5f-5d785bb49543" containerName="main" containerID="cri-o://6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e" gracePeriod=45
Nov 01 13:01:52.540174 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.540135    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c" podUID="1757ffa6-b479-45f0-bb5f-5d785bb49543" containerName="main" probeResult="failure" output="Get \"https://10.132.0.185:8000/ready\": dial tcp 10.132.0.185:8000: connect: connection refused"
Nov 01 13:01:52.616567 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.616534    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40cb0bff-44e1-409a-b7fe-891015b30fcb" path="/var/lib/kubelet/pods/40cb0bff-44e1-409a-b7fe-891015b30fcb/volumes"
Nov 01 13:01:52.616981 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.616967    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c585c6-9d66-4a3c-b13e-c6004c079867" path="/var/lib/kubelet/pods/a0c585c6-9d66-4a3c-b13e-c6004c079867/volumes"
Nov 01 13:01:52.650733 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.650674    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 13:01:52.764164 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764131    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-home\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-home\") pod \"1757ffa6-b479-45f0-bb5f-5d785bb49543\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") "
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764177    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1757ffa6-b479-45f0-bb5f-5d785bb49543-kube-api-access-rzmqz\" (UniqueName: \"kubernetes.io/projected/1757ffa6-b479-45f0-bb5f-5d785bb49543-kube-api-access-rzmqz\") pod \"1757ffa6-b479-45f0-bb5f-5d785bb49543\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") "
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764212    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-model-cache\") pod \"1757ffa6-b479-45f0-bb5f-5d785bb49543\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") "
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764280    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-tmp-dir\") pod \"1757ffa6-b479-45f0-bb5f-5d785bb49543\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") "
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764413    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-dshm\") pod \"1757ffa6-b479-45f0-bb5f-5d785bb49543\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") "
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764430    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-home" pod "1757ffa6-b479-45f0-bb5f-5d785bb49543" (UID: "1757ffa6-b479-45f0-bb5f-5d785bb49543"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764461    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1757ffa6-b479-45f0-bb5f-5d785bb49543-tls-certs\" (UniqueName: \"kubernetes.io/secret/1757ffa6-b479-45f0-bb5f-5d785bb49543-tls-certs\") pod \"1757ffa6-b479-45f0-bb5f-5d785bb49543\" (UID: \"1757ffa6-b479-45f0-bb5f-5d785bb49543\") "
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764753    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-tmp-dir" pod "1757ffa6-b479-45f0-bb5f-5d785bb49543" (UID: "1757ffa6-b479-45f0-bb5f-5d785bb49543"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764901    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764922    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:52.765144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.764977    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-model-cache" pod "1757ffa6-b479-45f0-bb5f-5d785bb49543" (UID: "1757ffa6-b479-45f0-bb5f-5d785bb49543"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:52.766946 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.766923    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1757ffa6-b479-45f0-bb5f-5d785bb49543-kube-api-access-rzmqz" pod "1757ffa6-b479-45f0-bb5f-5d785bb49543" (UID: "1757ffa6-b479-45f0-bb5f-5d785bb49543"). InnerVolumeSpecName "kube-api-access-rzmqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:52.767097 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.767033    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1757ffa6-b479-45f0-bb5f-5d785bb49543-tls-certs" pod "1757ffa6-b479-45f0-bb5f-5d785bb49543" (UID: "1757ffa6-b479-45f0-bb5f-5d785bb49543"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:52.767401 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.767370    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-dshm" pod "1757ffa6-b479-45f0-bb5f-5d785bb49543" (UID: "1757ffa6-b479-45f0-bb5f-5d785bb49543"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:52.866126 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.866090    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:52.866126 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.866119    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1757ffa6-b479-45f0-bb5f-5d785bb49543-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:52.866126 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.866129    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzmqz\" (UniqueName: \"kubernetes.io/projected/1757ffa6-b479-45f0-bb5f-5d785bb49543-kube-api-access-rzmqz\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:52.866376 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:52.866140    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1757ffa6-b479-45f0-bb5f-5d785bb49543-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:53.106123 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:53.106086    5725 generic.go:356] "Generic (PLEG): container finished" podID="1757ffa6-b479-45f0-bb5f-5d785bb49543" containerID="6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e" exitCode=0
Nov 01 13:01:53.106285 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:53.106158    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"
Nov 01 13:01:53.106285 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:53.106176    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c" event={"ID":"1757ffa6-b479-45f0-bb5f-5d785bb49543","Type":"ContainerDied","Data":"6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e"}
Nov 01 13:01:53.106285 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:53.106225    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c" event={"ID":"1757ffa6-b479-45f0-bb5f-5d785bb49543","Type":"ContainerDied","Data":"b0376e80066dd0764b38ee488ed2b112f739dd1d937ca017e849ec0db714634c"}
Nov 01 13:01:53.106285 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:53.106247    5725 scope.go:122] "RemoveContainer" containerID="6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e"
Nov 01 13:01:53.117224 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:53.117203    5725 scope.go:122] "RemoveContainer" containerID="6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e"
Nov 01 13:01:53.117530 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:01:53.117507    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e\": container with ID starting with 6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e not found: ID does not exist" containerID="6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e"
Nov 01 13:01:53.117610 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:53.117542    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e"} err="failed to get container status \"6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e\": rpc error: code = NotFound desc = could not find container \"6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e\": container with ID starting with 6db95e921372f2d5a329dff84f53eecbfafd2e9adf93501bbe25263f86cd603e not found: ID does not exist"
Nov 01 13:01:53.261623 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:53.261588    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"]
Nov 01 13:01:53.308248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:53.308215    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-187d06ed/e2e-lifecycle-model-187d06ed-kserve-6847cf4f57-mr86c"]
Nov 01 13:01:54.616583 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:54.616551    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1757ffa6-b479-45f0-bb5f-5d785bb49543" path="/var/lib/kubelet/pods/1757ffa6-b479-45f0-bb5f-5d785bb49543/volumes"
Nov 01 13:01:54.713819 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:54.713782    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:54.726487 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:54.726458    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:01:55.134357 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:01:55.134317    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:02:01.027013 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:02:01.026974    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:02:03.471778 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.471730    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"]
Nov 01 13:02:03.472507 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.472484    5725 cpu_manager.go:414] "containerMap: removing container" podUID="40cb0bff-44e1-409a-b7fe-891015b30fcb" containerName="payload-processing"
Nov 01 13:02:03.472613 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.472508    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40cb0bff-44e1-409a-b7fe-891015b30fcb" containerName="payload-processing"
Nov 01 13:02:03.472613 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.472561    5725 cpu_manager.go:414] "containerMap: removing container" podUID="a0c585c6-9d66-4a3c-b13e-c6004c079867" containerName="payload-pre-processing"
Nov 01 13:02:03.472613 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.472569    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c585c6-9d66-4a3c-b13e-c6004c079867" containerName="payload-pre-processing"
Nov 01 13:02:03.472613 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.472589    5725 cpu_manager.go:414] "containerMap: removing container" podUID="1757ffa6-b479-45f0-bb5f-5d785bb49543" containerName="main"
Nov 01 13:02:03.472613 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.472595    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1757ffa6-b479-45f0-bb5f-5d785bb49543" containerName="main"
Nov 01 13:02:03.472872 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.472716    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c585c6-9d66-4a3c-b13e-c6004c079867" containerName="payload-pre-processing"
Nov 01 13:02:03.472872 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.472731    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="1757ffa6-b479-45f0-bb5f-5d785bb49543" containerName="main"
Nov 01 13:02:03.472872 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.472748    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="40cb0bff-44e1-409a-b7fe-891015b30fcb" containerName="payload-processing"
Nov 01 13:02:03.477751 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.477726    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.528848 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.528812    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"]
Nov 01 13:02:03.544933 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.544901    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:02:03.572251 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.572211    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:02:03.584639 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.584602    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7079fd04-2dscr\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.584829 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.584746    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvxg\" (UniqueName: \"kubernetes.io/projected/1be8e5b0-7675-4c26-bf76-aab756c42786-kube-api-access-scvxg\") pod \"maas-api-revoke-sub-7079fd04-2dscr\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.584829 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.584776    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7079fd04-2dscr\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.685579 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.685539    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvxg\" (UniqueName: \"kubernetes.io/projected/1be8e5b0-7675-4c26-bf76-aab756c42786-kube-api-access-scvxg\") pod \"maas-api-revoke-sub-7079fd04-2dscr\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.685579 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.685579    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7079fd04-2dscr\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.685824 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.685690    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7079fd04-2dscr\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.686326 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.686298    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7079fd04-2dscr\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.688075 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.688042    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7079fd04-2dscr\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.739948 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.739865    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvxg\" (UniqueName: \"kubernetes.io/projected/1be8e5b0-7675-4c26-bf76-aab756c42786-kube-api-access-scvxg\") pod \"maas-api-revoke-sub-7079fd04-2dscr\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:03.792586 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:03.792545    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:04.191968 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:04.191885    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"]
Nov 01 13:02:04.198998 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:02:04.198954    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be8e5b0_7675_4c26_bf76_aab756c42786.slice/crio-2594dfcb66cf7e64e881dc394c0ab854a741678dcb50483ce2918d22387bb122 WatchSource:0}: Error finding container 2594dfcb66cf7e64e881dc394c0ab854a741678dcb50483ce2918d22387bb122: Status 404 returned error can't find the container with id 2594dfcb66cf7e64e881dc394c0ab854a741678dcb50483ce2918d22387bb122
Nov 01 13:02:05.205511 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:05.205473    5725 generic.go:356] "Generic (PLEG): container finished" podID="1be8e5b0-7675-4c26-bf76-aab756c42786" containerID="8ea6dc50ea559f1bda933f5bdcfde7afd56ff42519ef31b6519dbdc9c4abb89c" exitCode=0
Nov 01 13:02:05.206001 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:05.205556    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr" event={"ID":"1be8e5b0-7675-4c26-bf76-aab756c42786","Type":"ContainerDied","Data":"8ea6dc50ea559f1bda933f5bdcfde7afd56ff42519ef31b6519dbdc9c4abb89c"}
Nov 01 13:02:05.206001 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:05.205622    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr" event={"ID":"1be8e5b0-7675-4c26-bf76-aab756c42786","Type":"ContainerStarted","Data":"2594dfcb66cf7e64e881dc394c0ab854a741678dcb50483ce2918d22387bb122"}
Nov 01 13:02:06.389076 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.389036    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:06.519821 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.519778    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-openshift-ca\") pod \"1be8e5b0-7675-4c26-bf76-aab756c42786\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") "
Nov 01 13:02:06.520027 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.519955    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-xks-ca\") pod \"1be8e5b0-7675-4c26-bf76-aab756c42786\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") "
Nov 01 13:02:06.520124 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.520096    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be8e5b0-7675-4c26-bf76-aab756c42786-kube-api-access-scvxg\" (UniqueName: \"kubernetes.io/projected/1be8e5b0-7675-4c26-bf76-aab756c42786-kube-api-access-scvxg\") pod \"1be8e5b0-7675-4c26-bf76-aab756c42786\" (UID: \"1be8e5b0-7675-4c26-bf76-aab756c42786\") "
Nov 01 13:02:06.521894 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.521864    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-openshift-ca" pod "1be8e5b0-7675-4c26-bf76-aab756c42786" (UID: "1be8e5b0-7675-4c26-bf76-aab756c42786"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:06.524235 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.523928    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be8e5b0-7675-4c26-bf76-aab756c42786-kube-api-access-scvxg" pod "1be8e5b0-7675-4c26-bf76-aab756c42786" (UID: "1be8e5b0-7675-4c26-bf76-aab756c42786"). InnerVolumeSpecName "kube-api-access-scvxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:06.524617 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.524583    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-xks-ca" pod "1be8e5b0-7675-4c26-bf76-aab756c42786" (UID: "1be8e5b0-7675-4c26-bf76-aab756c42786"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:02:06.621304 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.621262    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:06.621304 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.621301    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvxg\" (UniqueName: \"kubernetes.io/projected/1be8e5b0-7675-4c26-bf76-aab756c42786-kube-api-access-scvxg\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:06.621585 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:06.621318    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be8e5b0-7675-4c26-bf76-aab756c42786-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:07.221621 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:07.221591    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"
Nov 01 13:02:07.221806 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:07.221621    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr" event={"ID":"1be8e5b0-7675-4c26-bf76-aab756c42786","Type":"ContainerDied","Data":"2594dfcb66cf7e64e881dc394c0ab854a741678dcb50483ce2918d22387bb122"}
Nov 01 13:02:07.221806 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:07.221681    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2594dfcb66cf7e64e881dc394c0ab854a741678dcb50483ce2918d22387bb122"
Nov 01 13:02:12.693576 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:12.693535    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"]
Nov 01 13:02:12.694144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:12.693803    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerName="istio-proxy" containerID="cri-o://a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a" gracePeriod=30
Nov 01 13:02:17.966970 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:17.966899    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 13:02:18.038189 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038155    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-socket\") pod \"bbbce536-40ca-49ee-b60d-0c2181800e4c\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") "
Nov 01 13:02:18.038394 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038242    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-token\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-token\") pod \"bbbce536-40ca-49ee-b60d-0c2181800e4c\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") "
Nov 01 13:02:18.038394 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038265    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-credential-socket\") pod \"bbbce536-40ca-49ee-b60d-0c2181800e4c\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") "
Nov 01 13:02:18.038394 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038285    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-envoy\") pod \"bbbce536-40ca-49ee-b60d-0c2181800e4c\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") "
Nov 01 13:02:18.038394 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038379    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbce536-40ca-49ee-b60d-0c2181800e4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbce536-40ca-49ee-b60d-0c2181800e4c-istiod-ca-cert\") pod \"bbbce536-40ca-49ee-b60d-0c2181800e4c\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") "
Nov 01 13:02:18.038644 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038421    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-certs\") pod \"bbbce536-40ca-49ee-b60d-0c2181800e4c\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") "
Nov 01 13:02:18.038644 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038493    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-data\") pod \"bbbce536-40ca-49ee-b60d-0c2181800e4c\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") "
Nov 01 13:02:18.038644 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038501    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-socket" pod "bbbce536-40ca-49ee-b60d-0c2181800e4c" (UID: "bbbce536-40ca-49ee-b60d-0c2181800e4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:18.038644 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038539    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-podinfo\") pod \"bbbce536-40ca-49ee-b60d-0c2181800e4c\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") "
Nov 01 13:02:18.038644 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038582    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-kube-api-access-lwnpb\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-kube-api-access-lwnpb\") pod \"bbbce536-40ca-49ee-b60d-0c2181800e4c\" (UID: \"bbbce536-40ca-49ee-b60d-0c2181800e4c\") "
Nov 01 13:02:18.038882 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038622    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-credential-socket" pod "bbbce536-40ca-49ee-b60d-0c2181800e4c" (UID: "bbbce536-40ca-49ee-b60d-0c2181800e4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:18.038882 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038724    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-data" pod "bbbce536-40ca-49ee-b60d-0c2181800e4c" (UID: "bbbce536-40ca-49ee-b60d-0c2181800e4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:18.038882 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038745    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-certs" pod "bbbce536-40ca-49ee-b60d-0c2181800e4c" (UID: "bbbce536-40ca-49ee-b60d-0c2181800e4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:18.038882 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.038802    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbce536-40ca-49ee-b60d-0c2181800e4c-istiod-ca-cert" pod "bbbce536-40ca-49ee-b60d-0c2181800e4c" (UID: "bbbce536-40ca-49ee-b60d-0c2181800e4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:18.039098 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.039023    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:18.039098 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.039061    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbce536-40ca-49ee-b60d-0c2181800e4c-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:18.039098 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.039076    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:18.039098 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.039089    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:18.039329 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.039102    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:18.040605 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.040582    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-podinfo" pod "bbbce536-40ca-49ee-b60d-0c2181800e4c" (UID: "bbbce536-40ca-49ee-b60d-0c2181800e4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:02:18.040605 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.040589    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-envoy" pod "bbbce536-40ca-49ee-b60d-0c2181800e4c" (UID: "bbbce536-40ca-49ee-b60d-0c2181800e4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:18.040764 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.040670    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-token" pod "bbbce536-40ca-49ee-b60d-0c2181800e4c" (UID: "bbbce536-40ca-49ee-b60d-0c2181800e4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:18.040764 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.040709    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-kube-api-access-lwnpb" pod "bbbce536-40ca-49ee-b60d-0c2181800e4c" (UID: "bbbce536-40ca-49ee-b60d-0c2181800e4c"). InnerVolumeSpecName "kube-api-access-lwnpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:18.140148 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.140116    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:18.140148 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.140143    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwnpb\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-kube-api-access-lwnpb\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:18.140148 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.140153    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:18.140390 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.140162    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbce536-40ca-49ee-b60d-0c2181800e4c-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:18.303584 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.303536    5725 generic.go:356] "Generic (PLEG): container finished" podID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerID="a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a" exitCode=0
Nov 01 13:02:18.303829 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.303598    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"
Nov 01 13:02:18.303829 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.303624    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" event={"ID":"bbbce536-40ca-49ee-b60d-0c2181800e4c","Type":"ContainerDied","Data":"a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a"}
Nov 01 13:02:18.303829 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.303683    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2" event={"ID":"bbbce536-40ca-49ee-b60d-0c2181800e4c","Type":"ContainerDied","Data":"83d20241b1e0f3218eac4972f0a4e3c993ffcb6b1dda7ec70ce9e2762bc74b2f"}
Nov 01 13:02:18.303829 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.303714    5725 scope.go:122] "RemoveContainer" containerID="a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a"
Nov 01 13:02:18.317614 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.317592    5725 scope.go:122] "RemoveContainer" containerID="a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a"
Nov 01 13:02:18.317975 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:02:18.317952    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a\": container with ID starting with a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a not found: ID does not exist" containerID="a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a"
Nov 01 13:02:18.318078 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.317985    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a"} err="failed to get container status \"a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a\": rpc error: code = NotFound desc = could not find container \"a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a\": container with ID starting with a540d4ecf1366a023067cc9140e7a8a1f5fa576d7c546d2bc20f5d6be9bc4b4a not found: ID does not exist"
Nov 01 13:02:18.459573 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.459533    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"]
Nov 01 13:02:18.466830 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.466797    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-jctd2"]
Nov 01 13:02:18.616062 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:18.615974    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" path="/var/lib/kubelet/pods/bbbce536-40ca-49ee-b60d-0c2181800e4c/volumes"
Nov 01 13:02:28.826704 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.826671    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"]
Nov 01 13:02:28.827246 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.827229    5725 cpu_manager.go:414] "containerMap: removing container" podUID="1be8e5b0-7675-4c26-bf76-aab756c42786" containerName="revoke-keys"
Nov 01 13:02:28.827246 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.827246    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be8e5b0-7675-4c26-bf76-aab756c42786" containerName="revoke-keys"
Nov 01 13:02:28.827340 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.827260    5725 cpu_manager.go:414] "containerMap: removing container" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerName="istio-proxy"
Nov 01 13:02:28.827340 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.827265    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerName="istio-proxy"
Nov 01 13:02:28.827404 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.827340    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be8e5b0-7675-4c26-bf76-aab756c42786" containerName="revoke-keys"
Nov 01 13:02:28.827404 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.827350    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbce536-40ca-49ee-b60d-0c2181800e4c" containerName="istio-proxy"
Nov 01 13:02:28.832314 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.832294    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:28.883954 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.883922    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"]
Nov 01 13:02:28.927230 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.927195    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b68322\"/\"kube-root-ca.crt\""
Nov 01 13:02:28.927230 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.927220    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b68322\"/\"openshift-service-ca.crt\""
Nov 01 13:02:28.927461 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.927250    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b68322\"/\"test-model-b68322-kserve-self-signed-certs\""
Nov 01 13:02:28.928869 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.928852    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b68322\"/\"default-dockercfg-k688q\""
Nov 01 13:02:28.968926 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.968893    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tmp-dir\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:28.969102 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.969000    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tls-certs\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:28.969178 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.969102    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-home\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:28.969178 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.969132    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5bx6\" (UniqueName: \"kubernetes.io/projected/c66adcd6-8708-444a-8781-0d1fee4e7fb8-kube-api-access-f5bx6\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:28.969261 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.969221    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-dshm\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:28.969308 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:28.969271    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-model-cache\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.070888 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.070847    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-home\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.070888 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.070888    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5bx6\" (UniqueName: \"kubernetes.io/projected/c66adcd6-8708-444a-8781-0d1fee4e7fb8-kube-api-access-f5bx6\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.071188 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.070958    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-dshm\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.071188 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.071002    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-model-cache\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.071188 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.071076    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tmp-dir\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.071188 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.071153    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tls-certs\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.071468 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.071444    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-home\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.071563 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.071540    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-model-cache\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.071623 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.071565    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tmp-dir\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.073446 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.073422    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-dshm\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.073762 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.073743    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tls-certs\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.126798 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.126722    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5bx6\" (UniqueName: \"kubernetes.io/projected/c66adcd6-8708-444a-8781-0d1fee4e7fb8-kube-api-access-f5bx6\") pod \"test-model-b68322-kserve-6cf9b7b84f-brwvc\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") " pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.144193 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.144161    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:29.525200 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:29.525162    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"]
Nov 01 13:02:29.532110 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:02:29.532080    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc66adcd6_8708_444a_8781_0d1fee4e7fb8.slice/crio-cf66ed30defc17b1c74233d81c2e0a3d8a795cd384b5578582eb7ba42174afbe WatchSource:0}: Error finding container cf66ed30defc17b1c74233d81c2e0a3d8a795cd384b5578582eb7ba42174afbe: Status 404 returned error can't find the container with id cf66ed30defc17b1c74233d81c2e0a3d8a795cd384b5578582eb7ba42174afbe
Nov 01 13:02:30.407386 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:30.407347    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc" event={"ID":"c66adcd6-8708-444a-8781-0d1fee4e7fb8","Type":"ContainerStarted","Data":"a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547"}
Nov 01 13:02:30.407386 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:30.407387    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc" event={"ID":"c66adcd6-8708-444a-8781-0d1fee4e7fb8","Type":"ContainerStarted","Data":"cf66ed30defc17b1c74233d81c2e0a3d8a795cd384b5578582eb7ba42174afbe"}
Nov 01 13:02:30.517984 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:30.517930    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc" podStartSLOduration=2.517915767 podStartE2EDuration="2.517915767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:02:30.514829832 +0000 UTC m=+2392.022021063" watchObservedRunningTime="2026-11-01 13:02:30.517915767 +0000 UTC m=+2392.025107002"
Nov 01 13:02:31.027530 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:02:31.027491    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:02:31.144918 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:31.144883    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:35.850238 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:35.850189    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"]
Nov 01 13:02:35.850883 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:35.850481    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy" containerID="cri-o://1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287" gracePeriod=30
Nov 01 13:02:35.853455 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:35.853426    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"]
Nov 01 13:02:35.853689 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:35.853653    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy" containerID="cri-o://88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7" gracePeriod=30
Nov 01 13:02:35.864614 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:35.864583    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"]
Nov 01 13:02:35.864825 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:35.864801    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy" containerID="cri-o://eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710" gracePeriod=30
Nov 01 13:02:36.104966 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:36.104881    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 13:02:36.105243 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:36.105211    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 13:02:36.155137 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:36.155099    5725 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 13:02:36.155324 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:36.155164    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 13:02:38.045628 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:38.045587    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"]
Nov 01 13:02:38.092791 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:38.092754    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec606b5-c8s4j"]
Nov 01 13:02:38.618239 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:38.618202    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279d8e5f-b758-42fa-acb6-af05ed4f382c" path="/var/lib/kubelet/pods/279d8e5f-b758-42fa-acb6-af05ed4f382c/volumes"
Nov 01 13:02:39.144326 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:39.144283    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:39.155601 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:39.155571    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:02:39.160494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:39.160467    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:39.165465 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:39.165436    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:02:39.178263 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:39.178243    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:02:39.186446 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:39.186426    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:02:39.509203 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:39.507342    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:02:41.159918 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.159574    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 13:02:41.199512 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.199479    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:02:41.202504 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.202485    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:02:41.211639 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211588    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-socket\") pod \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") "
Nov 01 13:02:41.211750 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211649    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-certs\") pod \"385cf49d-a997-42c7-9d2f-52c6dacdd881\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") "
Nov 01 13:02:41.211750 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211671    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-certs\") pod \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") "
Nov 01 13:02:41.211750 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211707    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-podinfo\") pod \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") "
Nov 01 13:02:41.211750 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211735    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-kube-api-access-mn82s\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-kube-api-access-mn82s\") pod \"385cf49d-a997-42c7-9d2f-52c6dacdd881\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") "
Nov 01 13:02:41.211989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211803    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-socket\") pod \"385cf49d-a997-42c7-9d2f-52c6dacdd881\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") "
Nov 01 13:02:41.211989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211856    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-data\") pod \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") "
Nov 01 13:02:41.211989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211874    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-socket" pod "1e4f885b-01d7-4675-9dff-d23d7f1499c1" (UID: "1e4f885b-01d7-4675-9dff-d23d7f1499c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.211989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211914    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-envoy\") pod \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") "
Nov 01 13:02:41.211989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211919    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-certs" pod "385cf49d-a997-42c7-9d2f-52c6dacdd881" (UID: "385cf49d-a997-42c7-9d2f-52c6dacdd881"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.211989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211954    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-token\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-token\") pod \"01083685-36fb-4dd7-9d26-3619c1cdef31\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") "
Nov 01 13:02:41.211989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.211980    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-podinfo\") pod \"385cf49d-a997-42c7-9d2f-52c6dacdd881\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") "
Nov 01 13:02:41.212396 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212011    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-envoy\") pod \"01083685-36fb-4dd7-9d26-3619c1cdef31\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") "
Nov 01 13:02:41.212396 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212028    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-certs" pod "1e4f885b-01d7-4675-9dff-d23d7f1499c1" (UID: "1e4f885b-01d7-4675-9dff-d23d7f1499c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.212396 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212036    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-podinfo\") pod \"01083685-36fb-4dd7-9d26-3619c1cdef31\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") "
Nov 01 13:02:41.212396 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212145    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-kube-api-access-jmzjk\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-kube-api-access-jmzjk\") pod \"01083685-36fb-4dd7-9d26-3619c1cdef31\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") "
Nov 01 13:02:41.212396 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212152    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-socket" pod "385cf49d-a997-42c7-9d2f-52c6dacdd881" (UID: "385cf49d-a997-42c7-9d2f-52c6dacdd881"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.212396 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212170    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-data" pod "1e4f885b-01d7-4675-9dff-d23d7f1499c1" (UID: "1e4f885b-01d7-4675-9dff-d23d7f1499c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.212396 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212186    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01083685-36fb-4dd7-9d26-3619c1cdef31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01083685-36fb-4dd7-9d26-3619c1cdef31-istiod-ca-cert\") pod \"01083685-36fb-4dd7-9d26-3619c1cdef31\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") "
Nov 01 13:02:41.212801 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212498    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01083685-36fb-4dd7-9d26-3619c1cdef31-istiod-ca-cert" pod "01083685-36fb-4dd7-9d26-3619c1cdef31" (UID: "01083685-36fb-4dd7-9d26-3619c1cdef31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:41.212801 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212688    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-token\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-token\") pod \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") "
Nov 01 13:02:41.212801 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212747    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-credential-socket\") pod \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") "
Nov 01 13:02:41.212976 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212802    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-certs\") pod \"01083685-36fb-4dd7-9d26-3619c1cdef31\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") "
Nov 01 13:02:41.212976 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212833    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-socket\") pod \"01083685-36fb-4dd7-9d26-3619c1cdef31\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") "
Nov 01 13:02:41.212976 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212890    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-credential-socket\") pod \"01083685-36fb-4dd7-9d26-3619c1cdef31\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") "
Nov 01 13:02:41.212976 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212929    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istiod-ca-cert\") pod \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") "
Nov 01 13:02:41.212976 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212959    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-credential-socket\") pod \"385cf49d-a997-42c7-9d2f-52c6dacdd881\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") "
Nov 01 13:02:41.213300 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.212997    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-data\") pod \"385cf49d-a997-42c7-9d2f-52c6dacdd881\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") "
Nov 01 13:02:41.213300 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213083    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-envoy\") pod \"385cf49d-a997-42c7-9d2f-52c6dacdd881\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") "
Nov 01 13:02:41.213300 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213147    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-token\") pod \"385cf49d-a997-42c7-9d2f-52c6dacdd881\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") "
Nov 01 13:02:41.213300 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213175    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/385cf49d-a997-42c7-9d2f-52c6dacdd881-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49d-a997-42c7-9d2f-52c6dacdd881-istiod-ca-cert\") pod \"385cf49d-a997-42c7-9d2f-52c6dacdd881\" (UID: \"385cf49d-a997-42c7-9d2f-52c6dacdd881\") "
Nov 01 13:02:41.213300 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213222    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-data\") pod \"01083685-36fb-4dd7-9d26-3619c1cdef31\" (UID: \"01083685-36fb-4dd7-9d26-3619c1cdef31\") "
Nov 01 13:02:41.213300 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213252    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-kube-api-access-8gc5h\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-kube-api-access-8gc5h\") pod \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\" (UID: \"1e4f885b-01d7-4675-9dff-d23d7f1499c1\") "
Nov 01 13:02:41.213735 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213706    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01083685-36fb-4dd7-9d26-3619c1cdef31-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.213830 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213739    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.213830 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213755    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.213830 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213769    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.213830 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213784    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.213830 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.213796    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.214697    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istiod-ca-cert" pod "1e4f885b-01d7-4675-9dff-d23d7f1499c1" (UID: "1e4f885b-01d7-4675-9dff-d23d7f1499c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.214721    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-credential-socket" pod "385cf49d-a997-42c7-9d2f-52c6dacdd881" (UID: "385cf49d-a997-42c7-9d2f-52c6dacdd881"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.214945    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-data" pod "385cf49d-a997-42c7-9d2f-52c6dacdd881" (UID: "385cf49d-a997-42c7-9d2f-52c6dacdd881"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.214947    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-credential-socket" pod "1e4f885b-01d7-4675-9dff-d23d7f1499c1" (UID: "1e4f885b-01d7-4675-9dff-d23d7f1499c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.215000    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-certs" pod "01083685-36fb-4dd7-9d26-3619c1cdef31" (UID: "01083685-36fb-4dd7-9d26-3619c1cdef31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.215017    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-data" pod "01083685-36fb-4dd7-9d26-3619c1cdef31" (UID: "01083685-36fb-4dd7-9d26-3619c1cdef31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.215039    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385cf49d-a997-42c7-9d2f-52c6dacdd881-istiod-ca-cert" pod "385cf49d-a997-42c7-9d2f-52c6dacdd881" (UID: "385cf49d-a997-42c7-9d2f-52c6dacdd881"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.215102    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-podinfo" pod "01083685-36fb-4dd7-9d26-3619c1cdef31" (UID: "01083685-36fb-4dd7-9d26-3619c1cdef31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.215104    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-socket" pod "01083685-36fb-4dd7-9d26-3619c1cdef31" (UID: "01083685-36fb-4dd7-9d26-3619c1cdef31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.215260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.215206    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-credential-socket" pod "01083685-36fb-4dd7-9d26-3619c1cdef31" (UID: "01083685-36fb-4dd7-9d26-3619c1cdef31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.215921 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.215509    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-podinfo" pod "1e4f885b-01d7-4675-9dff-d23d7f1499c1" (UID: "1e4f885b-01d7-4675-9dff-d23d7f1499c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:02:41.216493 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.216471    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-envoy" pod "01083685-36fb-4dd7-9d26-3619c1cdef31" (UID: "01083685-36fb-4dd7-9d26-3619c1cdef31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.216642 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.216495    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-token" pod "01083685-36fb-4dd7-9d26-3619c1cdef31" (UID: "01083685-36fb-4dd7-9d26-3619c1cdef31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.216642 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.216511    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-kube-api-access-jmzjk" pod "01083685-36fb-4dd7-9d26-3619c1cdef31" (UID: "01083685-36fb-4dd7-9d26-3619c1cdef31"). InnerVolumeSpecName "kube-api-access-jmzjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.216642 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.216544    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-podinfo" pod "385cf49d-a997-42c7-9d2f-52c6dacdd881" (UID: "385cf49d-a997-42c7-9d2f-52c6dacdd881"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:02:41.216835 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.216806    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-kube-api-access-mn82s" pod "385cf49d-a997-42c7-9d2f-52c6dacdd881" (UID: "385cf49d-a997-42c7-9d2f-52c6dacdd881"). InnerVolumeSpecName "kube-api-access-mn82s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.216835 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.216812    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-token" pod "385cf49d-a997-42c7-9d2f-52c6dacdd881" (UID: "385cf49d-a997-42c7-9d2f-52c6dacdd881"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.216929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.216906    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-envoy" pod "1e4f885b-01d7-4675-9dff-d23d7f1499c1" (UID: "1e4f885b-01d7-4675-9dff-d23d7f1499c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.216965 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.216921    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-kube-api-access-8gc5h" pod "1e4f885b-01d7-4675-9dff-d23d7f1499c1" (UID: "1e4f885b-01d7-4675-9dff-d23d7f1499c1"). InnerVolumeSpecName "kube-api-access-8gc5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.217311 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.217291    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-envoy" pod "385cf49d-a997-42c7-9d2f-52c6dacdd881" (UID: "385cf49d-a997-42c7-9d2f-52c6dacdd881"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.217692 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.217677    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-token" pod "1e4f885b-01d7-4675-9dff-d23d7f1499c1" (UID: "1e4f885b-01d7-4675-9dff-d23d7f1499c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.315158 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315119    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmzjk\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-kube-api-access-jmzjk\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315158 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315160    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315175    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315189    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315205    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315221    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315233    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315248    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315262    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315275    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315288    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315299    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cf49d-a997-42c7-9d2f-52c6dacdd881-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315311    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315325    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gc5h\" (UniqueName: \"kubernetes.io/projected/1e4f885b-01d7-4675-9dff-d23d7f1499c1-kube-api-access-8gc5h\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315338    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315358 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315351    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn82s\" (UniqueName: \"kubernetes.io/projected/385cf49d-a997-42c7-9d2f-52c6dacdd881-kube-api-access-mn82s\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315883 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315364    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4f885b-01d7-4675-9dff-d23d7f1499c1-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315883 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315377    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315883 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315388    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cf49d-a997-42c7-9d2f-52c6dacdd881-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315883 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315396    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.315883 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.315405    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01083685-36fb-4dd7-9d26-3619c1cdef31-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.499855 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.499745    5725 generic.go:356] "Generic (PLEG): container finished" podID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerID="88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7" exitCode=0
Nov 01 13:02:41.499855 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.499812    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" event={"ID":"385cf49d-a997-42c7-9d2f-52c6dacdd881","Type":"ContainerDied","Data":"88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7"}
Nov 01 13:02:41.500145 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.499872    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7" event={"ID":"385cf49d-a997-42c7-9d2f-52c6dacdd881","Type":"ContainerDied","Data":"c957021583f7a38ec1e081f72813931ca2bbd602b69451527b9f73cb2f9d900e"}
Nov 01 13:02:41.500145 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.499895    5725 scope.go:122] "RemoveContainer" containerID="88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7"
Nov 01 13:02:41.500145 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.499829    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"
Nov 01 13:02:41.502515 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.502490    5725 generic.go:356] "Generic (PLEG): container finished" podID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerID="eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710" exitCode=0
Nov 01 13:02:41.502682 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.502668    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"
Nov 01 13:02:41.503278 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.502890    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" event={"ID":"1e4f885b-01d7-4675-9dff-d23d7f1499c1","Type":"ContainerDied","Data":"eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710"}
Nov 01 13:02:41.503278 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.502967    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt" event={"ID":"1e4f885b-01d7-4675-9dff-d23d7f1499c1","Type":"ContainerDied","Data":"7af7dd205306ef2f92ee301eb47584e60ea89171474af1e88ef3044e38cdebdc"}
Nov 01 13:02:41.515088 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.511013    5725 generic.go:356] "Generic (PLEG): container finished" podID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerID="1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287" exitCode=0
Nov 01 13:02:41.515088 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.511180    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" event={"ID":"01083685-36fb-4dd7-9d26-3619c1cdef31","Type":"ContainerDied","Data":"1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287"}
Nov 01 13:02:41.515088 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.511227    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7" event={"ID":"01083685-36fb-4dd7-9d26-3619c1cdef31","Type":"ContainerDied","Data":"0da4112042e225b42a87508716cfbd28158a4c06a1a2b22e73e9b74e9f0fefb1"}
Nov 01 13:02:41.515088 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.511319    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"
Nov 01 13:02:41.517004 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.516989    5725 scope.go:122] "RemoveContainer" containerID="91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff"
Nov 01 13:02:41.530296 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.530270    5725 scope.go:122] "RemoveContainer" containerID="88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7"
Nov 01 13:02:41.530651 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:02:41.530624    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7\": container with ID starting with 88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7 not found: ID does not exist" containerID="88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7"
Nov 01 13:02:41.530726 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.530667    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7"} err="failed to get container status \"88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7\": rpc error: code = NotFound desc = could not find container \"88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7\": container with ID starting with 88934c71114957132c85594e6340780449c7f40376377e84c80b5ca9f107eff7 not found: ID does not exist"
Nov 01 13:02:41.530726 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.530693    5725 scope.go:122] "RemoveContainer" containerID="91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff"
Nov 01 13:02:41.530998 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:02:41.530974    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff\": container with ID starting with 91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff not found: ID does not exist" containerID="91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff"
Nov 01 13:02:41.531072 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.531009    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff"} err="failed to get container status \"91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff\": rpc error: code = NotFound desc = could not find container \"91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff\": container with ID starting with 91ee671d9c541eaefaa906cd61d51e799917137822bee3ca544a85df5a8632ff not found: ID does not exist"
Nov 01 13:02:41.531072 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.531032    5725 scope.go:122] "RemoveContainer" containerID="eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710"
Nov 01 13:02:41.543269 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.543243    5725 scope.go:122] "RemoveContainer" containerID="eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710"
Nov 01 13:02:41.543769 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:02:41.543591    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710\": container with ID starting with eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710 not found: ID does not exist" containerID="eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710"
Nov 01 13:02:41.543769 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.543635    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710"} err="failed to get container status \"eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710\": rpc error: code = NotFound desc = could not find container \"eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710\": container with ID starting with eb5487bd2ae29bd9395b7d547f4adf8a35e9c3f39d4d6771be932c9b17ed2710 not found: ID does not exist"
Nov 01 13:02:41.543769 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.543654    5725 scope.go:122] "RemoveContainer" containerID="1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287"
Nov 01 13:02:41.554062 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.554031    5725 scope.go:122] "RemoveContainer" containerID="1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287"
Nov 01 13:02:41.554471 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:02:41.554447    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287\": container with ID starting with 1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287 not found: ID does not exist" containerID="1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287"
Nov 01 13:02:41.554579 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.554480    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287"} err="failed to get container status \"1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287\": rpc error: code = NotFound desc = could not find container \"1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287\": container with ID starting with 1fddd8452aa1bf0f2e2c15bf3a3ee0bfa4c0dbfb4d7d8fb92791b99031daf287 not found: ID does not exist"
Nov 01 13:02:41.656979 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.656942    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"]
Nov 01 13:02:41.663589 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.663553    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-jgwm7"]
Nov 01 13:02:41.766792 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.766708    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"]
Nov 01 13:02:41.774354 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.774326    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-86pjt"]
Nov 01 13:02:41.878735 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.878698    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"]
Nov 01 13:02:41.883312 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:41.883285    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-kx8q7"]
Nov 01 13:02:42.616319 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:42.616284    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" path="/var/lib/kubelet/pods/01083685-36fb-4dd7-9d26-3619c1cdef31/volumes"
Nov 01 13:02:42.616750 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:42.616736    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" path="/var/lib/kubelet/pods/1e4f885b-01d7-4675-9dff-d23d7f1499c1/volumes"
Nov 01 13:02:42.617148 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:42.617134    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" path="/var/lib/kubelet/pods/385cf49d-a997-42c7-9d2f-52c6dacdd881/volumes"
Nov 01 13:02:52.071592 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:52.071103    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"]
Nov 01 13:02:52.075386 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:52.075333    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b51fa788-2lwc8"]
Nov 01 13:02:52.615568 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:52.615532    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88057d9b-59f2-4e5f-8f1c-8057398cd2f9" path="/var/lib/kubelet/pods/88057d9b-59f2-4e5f-8f1c-8057398cd2f9/volumes"
Nov 01 13:02:54.602072 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.601287    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"]
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602111    5725 cpu_manager.go:414] "containerMap: removing container" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602133    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602156    5725 cpu_manager.go:414] "containerMap: removing container" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602164    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602189    5725 cpu_manager.go:414] "containerMap: removing container" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602196    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602222    5725 cpu_manager.go:414] "containerMap: removing container" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602229    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602344    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602358    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="01083685-36fb-4dd7-9d26-3619c1cdef31" containerName="istio-proxy"
Nov 01 13:02:54.602603 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.602371    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4f885b-01d7-4675-9dff-d23d7f1499c1" containerName="istio-proxy"
Nov 01 13:02:54.641071 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.640944    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.730355 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.730320    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"]
Nov 01 13:02:54.813512 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.813467    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.813682 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.813524    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.813682 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.813645    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63317aed-7a2c-41fe-8eee-ece29d7326cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.813799 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.813768    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkzx\" (UniqueName: \"kubernetes.io/projected/63317aed-7a2c-41fe-8eee-ece29d7326cc-kube-api-access-9nkzx\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.813845 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.813828    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.813928 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.813912    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.813988 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.813943    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.813988 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.813980    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.814109 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.814093    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915112 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915018    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915112 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915077    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915112 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915105    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915362 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915136    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63317aed-7a2c-41fe-8eee-ece29d7326cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915362 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915187    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkzx\" (UniqueName: \"kubernetes.io/projected/63317aed-7a2c-41fe-8eee-ece29d7326cc-kube-api-access-9nkzx\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915362 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915227    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915362 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915289    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915362 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915319    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915362 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915346    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915645 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915514    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915645 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915573    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.915767 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.915689    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.916277 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.916234    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63317aed-7a2c-41fe-8eee-ece29d7326cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.916407 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.916306    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.917740 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.917720    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.918011 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.917991    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.969476 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.969442    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63317aed-7a2c-41fe-8eee-ece29d7326cc-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:54.969770 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:54.969742    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkzx\" (UniqueName: \"kubernetes.io/projected/63317aed-7a2c-41fe-8eee-ece29d7326cc-kube-api-access-9nkzx\") pod \"maas-default-gateway-openshift-default-7df854bf67-rxn6r\" (UID: \"63317aed-7a2c-41fe-8eee-ece29d7326cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:55.267010 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:55.266973    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:55.715220 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:55.715133    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"]
Nov 01 13:02:55.758171 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:02:55.758141    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63317aed_7a2c_41fe_8eee_ece29d7326cc.slice/crio-70c473c2dc6f9825d65367d10b8fa6c177a39d61dc19acb5ed5c39526731985c WatchSource:0}: Error finding container 70c473c2dc6f9825d65367d10b8fa6c177a39d61dc19acb5ed5c39526731985c: Status 404 returned error can't find the container with id 70c473c2dc6f9825d65367d10b8fa6c177a39d61dc19acb5ed5c39526731985c
Nov 01 13:02:55.760544 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:55.760508    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:02:55.760682 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:55.760579    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:02:55.760682 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:55.760608    5725 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 13:02:56.661233 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:56.661192    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" event={"ID":"63317aed-7a2c-41fe-8eee-ece29d7326cc","Type":"ContainerStarted","Data":"c46e496fbda88cae7d8cf1eadf44760177510115ab28d4f63457acc396066e11"}
Nov 01 13:02:56.661446 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:56.661244    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" event={"ID":"63317aed-7a2c-41fe-8eee-ece29d7326cc","Type":"ContainerStarted","Data":"70c473c2dc6f9825d65367d10b8fa6c177a39d61dc19acb5ed5c39526731985c"}
Nov 01 13:02:56.825409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:56.825353    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" podStartSLOduration=2.825338122 podStartE2EDuration="2.825338122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:02:56.82286533 +0000 UTC m=+2418.330056547" watchObservedRunningTime="2026-11-01 13:02:56.825338122 +0000 UTC m=+2418.332529339"
Nov 01 13:02:57.267363 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:57.267333    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:02:57.268460 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:57.268440    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rxn6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 01 13:02:57.268526 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:57.268484    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" podUID="63317aed-7a2c-41fe-8eee-ece29d7326cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 01 13:02:58.268417 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:58.268379    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rxn6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 01 13:02:58.268838 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:58.268455    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" podUID="63317aed-7a2c-41fe-8eee-ece29d7326cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 01 13:02:59.268323 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:59.268284    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rxn6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 01 13:02:59.268545 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:02:59.268362    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" podUID="63317aed-7a2c-41fe-8eee-ece29d7326cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 01 13:03:00.268063 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:00.268026    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rxn6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 01 13:03:00.268240 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:00.268090    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" podUID="63317aed-7a2c-41fe-8eee-ece29d7326cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 01 13:03:01.045075 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:03:01.038235    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:03:01.267856 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:01.267817    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rxn6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 01 13:03:01.268035 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:01.267886    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" podUID="63317aed-7a2c-41fe-8eee-ece29d7326cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 01 13:03:02.267600 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:02.267560    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rxn6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 01 13:03:02.268038 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:02.267637    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" podUID="63317aed-7a2c-41fe-8eee-ece29d7326cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 01 13:03:03.268121 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:03.268084    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rxn6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Nov 01 13:03:03.268509 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:03.268153    5725 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" podUID="63317aed-7a2c-41fe-8eee-ece29d7326cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Nov 01 13:03:04.317776 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:04.317721    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:03:04.318286 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:04.318131    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:03:04.319200 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:04.319173    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r"
Nov 01 13:03:31.027485 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:03:31.027395    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:03:41.924197 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:41.924159    5725 scope.go:122] "RemoveContainer" containerID="4e5b33bf3fd0638a5b2a8751bc4b3fa49a9cf93874320942e028403fc7dc15e0"
Nov 01 13:03:49.353943 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:49.353897    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"]
Nov 01 13:03:49.401269 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:49.400884    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"]
Nov 01 13:03:56.584194 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:03:56.584153    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"]
Nov 01 13:04:01.026637 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:04:01.026604    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:04:01.585212 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:01.585161    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv" podUID="2cc6f9cc-7039-4174-967c-d264de6c076e" containerName="payload-pre-processing" containerID="cri-o://da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43" gracePeriod=25
Nov 01 13:04:04.374422 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.374378    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" containerID="cri-o://573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948" gracePeriod=45
Nov 01 13:04:04.430510 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.430461    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main" containerID="cri-o://56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057" gracePeriod=45
Nov 01 13:04:04.705859 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.705832    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 13:04:04.740970 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.740944    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 13:04:04.808704 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.808677    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tls-certs\") pod \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") "
Nov 01 13:04:04.808960 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.808726    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8dec32-afbe-4385-a7e9-d57fc37848ac-kube-api-access-mndlr\" (UniqueName: \"kubernetes.io/projected/cd8dec32-afbe-4385-a7e9-d57fc37848ac-kube-api-access-mndlr\") pod \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") "
Nov 01 13:04:04.808960 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.808761    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-home\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-home\") pod \"77084dd3-4b8d-4314-925d-478db9af9706\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") "
Nov 01 13:04:04.808960 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.808799    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77084dd3-4b8d-4314-925d-478db9af9706-kube-api-access-86l9l\" (UniqueName: \"kubernetes.io/projected/77084dd3-4b8d-4314-925d-478db9af9706-kube-api-access-86l9l\") pod \"77084dd3-4b8d-4314-925d-478db9af9706\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") "
Nov 01 13:04:04.808960 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.808835    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-dshm\") pod \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") "
Nov 01 13:04:04.808960 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.808894    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-home\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-home\") pod \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") "
Nov 01 13:04:04.808960 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.808941    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-model-cache\") pod \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") "
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.808978    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-model-cache\") pod \"77084dd3-4b8d-4314-925d-478db9af9706\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") "
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.809014    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tmp-dir\") pod \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\" (UID: \"cd8dec32-afbe-4385-a7e9-d57fc37848ac\") "
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.809064    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-dshm\") pod \"77084dd3-4b8d-4314-925d-478db9af9706\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") "
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.809111    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77084dd3-4b8d-4314-925d-478db9af9706-tls-certs\" (UniqueName: \"kubernetes.io/secret/77084dd3-4b8d-4314-925d-478db9af9706-tls-certs\") pod \"77084dd3-4b8d-4314-925d-478db9af9706\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") "
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.809154    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-tmp-dir\") pod \"77084dd3-4b8d-4314-925d-478db9af9706\" (UID: \"77084dd3-4b8d-4314-925d-478db9af9706\") "
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.809516    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-home" pod "77084dd3-4b8d-4314-925d-478db9af9706" (UID: "77084dd3-4b8d-4314-925d-478db9af9706"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.809470    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-model-cache" pod "cd8dec32-afbe-4385-a7e9-d57fc37848ac" (UID: "cd8dec32-afbe-4385-a7e9-d57fc37848ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.809763    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-model-cache" pod "77084dd3-4b8d-4314-925d-478db9af9706" (UID: "77084dd3-4b8d-4314-925d-478db9af9706"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.809988    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tmp-dir" pod "cd8dec32-afbe-4385-a7e9-d57fc37848ac" (UID: "cd8dec32-afbe-4385-a7e9-d57fc37848ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.810447 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.810079    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-home" pod "cd8dec32-afbe-4385-a7e9-d57fc37848ac" (UID: "cd8dec32-afbe-4385-a7e9-d57fc37848ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.811112 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.811078    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-tmp-dir" pod "77084dd3-4b8d-4314-925d-478db9af9706" (UID: "77084dd3-4b8d-4314-925d-478db9af9706"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.811576 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.811539    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tls-certs" pod "cd8dec32-afbe-4385-a7e9-d57fc37848ac" (UID: "cd8dec32-afbe-4385-a7e9-d57fc37848ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:04:04.811907 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.811871    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-dshm" pod "cd8dec32-afbe-4385-a7e9-d57fc37848ac" (UID: "cd8dec32-afbe-4385-a7e9-d57fc37848ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.812135 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.812114    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8dec32-afbe-4385-a7e9-d57fc37848ac-kube-api-access-mndlr" pod "cd8dec32-afbe-4385-a7e9-d57fc37848ac" (UID: "cd8dec32-afbe-4385-a7e9-d57fc37848ac"). InnerVolumeSpecName "kube-api-access-mndlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:04:04.812317 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.812293    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77084dd3-4b8d-4314-925d-478db9af9706-kube-api-access-86l9l" pod "77084dd3-4b8d-4314-925d-478db9af9706" (UID: "77084dd3-4b8d-4314-925d-478db9af9706"). InnerVolumeSpecName "kube-api-access-86l9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:04:04.812852 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.812837    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-dshm" pod "77084dd3-4b8d-4314-925d-478db9af9706" (UID: "77084dd3-4b8d-4314-925d-478db9af9706"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.813248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.813235    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77084dd3-4b8d-4314-925d-478db9af9706-tls-certs" pod "77084dd3-4b8d-4314-925d-478db9af9706" (UID: "77084dd3-4b8d-4314-925d-478db9af9706"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:04:04.910697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910614    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.910697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910644    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.910697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910654    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77084dd3-4b8d-4314-925d-478db9af9706-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.910697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910662    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.910697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910671    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd8dec32-afbe-4385-a7e9-d57fc37848ac-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.910697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910682    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mndlr\" (UniqueName: \"kubernetes.io/projected/cd8dec32-afbe-4385-a7e9-d57fc37848ac-kube-api-access-mndlr\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.910697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910692    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.910697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910701    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86l9l\" (UniqueName: \"kubernetes.io/projected/77084dd3-4b8d-4314-925d-478db9af9706-kube-api-access-86l9l\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.911085 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910709    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.911085 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910718    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.911085 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910725    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd8dec32-afbe-4385-a7e9-d57fc37848ac-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.911085 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:04.910734    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77084dd3-4b8d-4314-925d-478db9af9706-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:05.197635 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.197545    5725 generic.go:356] "Generic (PLEG): container finished" podID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerID="56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057" exitCode=0
Nov 01 13:04:05.197635 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.197602    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"
Nov 01 13:04:05.197635 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.197624    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" event={"ID":"cd8dec32-afbe-4385-a7e9-d57fc37848ac","Type":"ContainerDied","Data":"56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057"}
Nov 01 13:04:05.197929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.197670    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4" event={"ID":"cd8dec32-afbe-4385-a7e9-d57fc37848ac","Type":"ContainerDied","Data":"63570572789b83e41ed257f71ad0431961d384a0bfa6a042cc8f73c524ee48ec"}
Nov 01 13:04:05.197929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.197688    5725 scope.go:122] "RemoveContainer" containerID="56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057"
Nov 01 13:04:05.199207 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.199180    5725 generic.go:356] "Generic (PLEG): container finished" podID="77084dd3-4b8d-4314-925d-478db9af9706" containerID="573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948" exitCode=0
Nov 01 13:04:05.199347 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.199228    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"
Nov 01 13:04:05.199347 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.199237    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" event={"ID":"77084dd3-4b8d-4314-925d-478db9af9706","Type":"ContainerDied","Data":"573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948"}
Nov 01 13:04:05.199347 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.199273    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" event={"ID":"77084dd3-4b8d-4314-925d-478db9af9706","Type":"ContainerDied","Data":"8810056f8b67dc46a6f0af0c6c726143644d4372c0b5b9cfbeda06ee5932499c"}
Nov 01 13:04:05.207816 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.207797    5725 scope.go:122] "RemoveContainer" containerID="56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057"
Nov 01 13:04:05.208190 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:04:05.208164    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057\": container with ID starting with 56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057 not found: ID does not exist" containerID="56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057"
Nov 01 13:04:05.208326 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.208201    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057"} err="failed to get container status \"56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057\": rpc error: code = NotFound desc = could not find container \"56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057\": container with ID starting with 56a2ed2dc0083f6cb5b16ca4a6bdf59f0048d9c48a22fe0c779ec8af30007057 not found: ID does not exist"
Nov 01 13:04:05.208326 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.208227    5725 scope.go:122] "RemoveContainer" containerID="573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948"
Nov 01 13:04:05.217821 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.217805    5725 scope.go:122] "RemoveContainer" containerID="573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948"
Nov 01 13:04:05.218167 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:04:05.218148    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948\": container with ID starting with 573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948 not found: ID does not exist" containerID="573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948"
Nov 01 13:04:05.218222 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.218176    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948"} err="failed to get container status \"573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948\": rpc error: code = NotFound desc = could not find container \"573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948\": container with ID starting with 573403418dcab9e9e383a050afde51791658ae3b00a13b6353d59059e9522948 not found: ID does not exist"
Nov 01 13:04:05.352924 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.352890    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"]
Nov 01 13:04:05.358746 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.358719    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t"]
Nov 01 13:04:05.460917 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.460830    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"]
Nov 01 13:04:05.465852 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.465825    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/simulator-w4-06fd6b-kserve-66468c4dbc-l8qm4"]
Nov 01 13:04:05.688332 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:05.688296    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-2-w4-06fd6b-kserve-75f74748f4-xb85t" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main" probeResult="failure" output="Get \"https://10.132.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:04:06.616756 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:06.616714    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77084dd3-4b8d-4314-925d-478db9af9706" path="/var/lib/kubelet/pods/77084dd3-4b8d-4314-925d-478db9af9706/volumes"
Nov 01 13:04:06.617417 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:06.617393    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" path="/var/lib/kubelet/pods/cd8dec32-afbe-4385-a7e9-d57fc37848ac/volumes"
Nov 01 13:04:08.431078 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:08.431024    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv" podUID="2cc6f9cc-7039-4174-967c-d264de6c076e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:04:18.430940 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:18.430900    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv" podUID="2cc6f9cc-7039-4174-967c-d264de6c076e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:04:21.845627 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:21.845564    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 13:04:22.003278 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.003237    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc6f9cc-7039-4174-967c-d264de6c076e-kube-api-access-gtr4q\" (UniqueName: \"kubernetes.io/projected/2cc6f9cc-7039-4174-967c-d264de6c076e-kube-api-access-gtr4q\") pod \"2cc6f9cc-7039-4174-967c-d264de6c076e\" (UID: \"2cc6f9cc-7039-4174-967c-d264de6c076e\") "
Nov 01 13:04:22.003481 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.003294    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc6f9cc-7039-4174-967c-d264de6c076e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc6f9cc-7039-4174-967c-d264de6c076e-plugins-config-volume\") pod \"2cc6f9cc-7039-4174-967c-d264de6c076e\" (UID: \"2cc6f9cc-7039-4174-967c-d264de6c076e\") "
Nov 01 13:04:22.003686 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.003660    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc6f9cc-7039-4174-967c-d264de6c076e-plugins-config-volume" pod "2cc6f9cc-7039-4174-967c-d264de6c076e" (UID: "2cc6f9cc-7039-4174-967c-d264de6c076e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:04:22.005343 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.005313    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc6f9cc-7039-4174-967c-d264de6c076e-kube-api-access-gtr4q" pod "2cc6f9cc-7039-4174-967c-d264de6c076e" (UID: "2cc6f9cc-7039-4174-967c-d264de6c076e"). InnerVolumeSpecName "kube-api-access-gtr4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:04:22.104988 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.104910    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtr4q\" (UniqueName: \"kubernetes.io/projected/2cc6f9cc-7039-4174-967c-d264de6c076e-kube-api-access-gtr4q\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:22.104988 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.104937    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc6f9cc-7039-4174-967c-d264de6c076e-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:22.320664 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.320622    5725 generic.go:356] "Generic (PLEG): container finished" podID="2cc6f9cc-7039-4174-967c-d264de6c076e" containerID="da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43" exitCode=0
Nov 01 13:04:22.320871 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.320680    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"
Nov 01 13:04:22.320871 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.320714    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv" event={"ID":"2cc6f9cc-7039-4174-967c-d264de6c076e","Type":"ContainerDied","Data":"da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43"}
Nov 01 13:04:22.320871 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.320801    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv" event={"ID":"2cc6f9cc-7039-4174-967c-d264de6c076e","Type":"ContainerDied","Data":"b4f278f4b971f2a5ae5072a6b51528304eb961ae2d0bc70cc3e20653510d9f88"}
Nov 01 13:04:22.320871 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.320831    5725 scope.go:122] "RemoveContainer" containerID="da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43"
Nov 01 13:04:22.331007 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.330991    5725 scope.go:122] "RemoveContainer" containerID="da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43"
Nov 01 13:04:22.331318 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:04:22.331299    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43\": container with ID starting with da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43 not found: ID does not exist" containerID="da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43"
Nov 01 13:04:22.331387 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.331328    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43"} err="failed to get container status \"da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43\": rpc error: code = NotFound desc = could not find container \"da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43\": container with ID starting with da40a269ecfe3d1268b2b4757b3818ad9fd0279d399f5d2cfe4dfd8b443f2b43 not found: ID does not exist"
Nov 01 13:04:22.476523 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.476495    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"]
Nov 01 13:04:22.478860 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.478833    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-06fd6b-648c5df555-mb7nv"]
Nov 01 13:04:22.616283 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:22.616241    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc6f9cc-7039-4174-967c-d264de6c076e" path="/var/lib/kubelet/pods/2cc6f9cc-7039-4174-967c-d264de6c076e/volumes"
Nov 01 13:04:31.031165 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:04:31.031130    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:04:41.949403 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:41.949371    5725 scope.go:122] "RemoveContainer" containerID="ab29f46c8df58eef29c1e68f6ac1cd8f67047441b37a8f37b4538e12e9d766c4"
Nov 01 13:04:41.959269 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:41.959251    5725 scope.go:122] "RemoveContainer" containerID="56067a4859d3c508da5477fad0afa812b5c6cb5facdc56e470f2d5249a7c8c5b"
Nov 01 13:04:41.968221 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:41.968196    5725 scope.go:122] "RemoveContainer" containerID="e0a06ce363abcbf19a6e162846a5e766a6b663a9be8c59a4536030f38363d41d"
Nov 01 13:04:41.976749 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:04:41.976734    5725 scope.go:122] "RemoveContainer" containerID="86b2338c6cfb8e0e2ad208f1f9d8e0c0490dd8f8bc9d628a19f781f4653d817a"
Nov 01 13:05:01.027841 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:05:01.027762    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:05:10.393185 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:10.393144    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"]
Nov 01 13:05:21.046315 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:21.046272    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"]
Nov 01 13:05:21.091572 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:21.091533    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112d13a5-glqgl"]
Nov 01 13:05:22.615822 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:22.615788    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fecb164-8181-44b0-9264-8d18ca7f1f60" path="/var/lib/kubelet/pods/7fecb164-8181-44b0-9264-8d18ca7f1f60/volumes"
Nov 01 13:05:25.413392 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.413349    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk" podUID="d25850a7-529b-41db-8318-054a24043526" containerName="main" containerID="cri-o://b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167" gracePeriod=45
Nov 01 13:05:25.677082 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.677004    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:05:25.755963 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.755921    5725 generic.go:356] "Generic (PLEG): container finished" podID="d25850a7-529b-41db-8318-054a24043526" containerID="b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167" exitCode=0
Nov 01 13:05:25.756294 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.756268    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk" event={"ID":"d25850a7-529b-41db-8318-054a24043526","Type":"ContainerDied","Data":"b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167"}
Nov 01 13:05:25.756435 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.756420    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk" event={"ID":"d25850a7-529b-41db-8318-054a24043526","Type":"ContainerDied","Data":"e8d1dccbc2d346390377a0b1a5bfee8d99ed8cd3776e43d55cbea40924f1ea79"}
Nov 01 13:05:25.756551 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.756538    5725 scope.go:122] "RemoveContainer" containerID="b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167"
Nov 01 13:05:25.757132 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.757098    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"
Nov 01 13:05:25.769566 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.769538    5725 scope.go:122] "RemoveContainer" containerID="b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167"
Nov 01 13:05:25.769883 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:05:25.769852    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167\": container with ID starting with b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167 not found: ID does not exist" containerID="b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167"
Nov 01 13:05:25.769945 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.769896    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167"} err="failed to get container status \"b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167\": rpc error: code = NotFound desc = could not find container \"b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167\": container with ID starting with b01922bdd58f7e1e310367fcd8f26f5ed255686fd93f2efe206e76bc79daa167 not found: ID does not exist"
Nov 01 13:05:25.781309 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781284    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25850a7-529b-41db-8318-054a24043526-tls-certs\" (UniqueName: \"kubernetes.io/secret/d25850a7-529b-41db-8318-054a24043526-tls-certs\") pod \"d25850a7-529b-41db-8318-054a24043526\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") "
Nov 01 13:05:25.781412 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781340    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-model-cache\") pod \"d25850a7-529b-41db-8318-054a24043526\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") "
Nov 01 13:05:25.781465 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781430    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-tmp-dir\") pod \"d25850a7-529b-41db-8318-054a24043526\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") "
Nov 01 13:05:25.781465 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781454    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-dshm\") pod \"d25850a7-529b-41db-8318-054a24043526\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") "
Nov 01 13:05:25.781560 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781475    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-home\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-home\") pod \"d25850a7-529b-41db-8318-054a24043526\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") "
Nov 01 13:05:25.781560 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781496    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25850a7-529b-41db-8318-054a24043526-kube-api-access-z5smh\" (UniqueName: \"kubernetes.io/projected/d25850a7-529b-41db-8318-054a24043526-kube-api-access-z5smh\") pod \"d25850a7-529b-41db-8318-054a24043526\" (UID: \"d25850a7-529b-41db-8318-054a24043526\") "
Nov 01 13:05:25.781677 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781597    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-model-cache" pod "d25850a7-529b-41db-8318-054a24043526" (UID: "d25850a7-529b-41db-8318-054a24043526"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:25.781761 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781736    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-tmp-dir" pod "d25850a7-529b-41db-8318-054a24043526" (UID: "d25850a7-529b-41db-8318-054a24043526"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:25.781822 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781755    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-home" pod "d25850a7-529b-41db-8318-054a24043526" (UID: "d25850a7-529b-41db-8318-054a24043526"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:25.781892 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781875    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:25.781937 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781897    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:25.781937 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.781911    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:25.783588 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.783565    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25850a7-529b-41db-8318-054a24043526-tls-certs" pod "d25850a7-529b-41db-8318-054a24043526" (UID: "d25850a7-529b-41db-8318-054a24043526"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:05:25.783588 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.783579    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-dshm" pod "d25850a7-529b-41db-8318-054a24043526" (UID: "d25850a7-529b-41db-8318-054a24043526"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:25.783721 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.783607    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25850a7-529b-41db-8318-054a24043526-kube-api-access-z5smh" pod "d25850a7-529b-41db-8318-054a24043526" (UID: "d25850a7-529b-41db-8318-054a24043526"). InnerVolumeSpecName "kube-api-access-z5smh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:05:25.883338 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.883303    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25850a7-529b-41db-8318-054a24043526-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:25.883338 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.883338    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5smh\" (UniqueName: \"kubernetes.io/projected/d25850a7-529b-41db-8318-054a24043526-kube-api-access-z5smh\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:25.883338 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:25.883352    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25850a7-529b-41db-8318-054a24043526-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:26.174524 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:26.174482    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"]
Nov 01 13:05:26.219828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:26.219794    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-72078a/test-model-72078a-kserve-8696584c7c-jkghk"]
Nov 01 13:05:26.615718 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:26.615683    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25850a7-529b-41db-8318-054a24043526" path="/var/lib/kubelet/pods/d25850a7-529b-41db-8318-054a24043526/volumes"
Nov 01 13:05:31.028242 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:05:31.028205    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:05:37.214742 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:37.214701    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"]
Nov 01 13:05:52.234568 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.234525    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc" podUID="c66adcd6-8708-444a-8781-0d1fee4e7fb8" containerName="main" containerID="cri-o://a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547" gracePeriod=45
Nov 01 13:05:52.482570 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.482544    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:05:52.555102 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.554983    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tmp-dir\") pod \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") "
Nov 01 13:05:52.555102 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555026    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-model-cache\") pod \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") "
Nov 01 13:05:52.555359 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555168    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-home\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-home\") pod \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") "
Nov 01 13:05:52.555359 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555197    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tls-certs\") pod \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") "
Nov 01 13:05:52.555359 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555230    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-dshm\") pod \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") "
Nov 01 13:05:52.555359 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555297    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66adcd6-8708-444a-8781-0d1fee4e7fb8-kube-api-access-f5bx6\" (UniqueName: \"kubernetes.io/projected/c66adcd6-8708-444a-8781-0d1fee4e7fb8-kube-api-access-f5bx6\") pod \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\" (UID: \"c66adcd6-8708-444a-8781-0d1fee4e7fb8\") "
Nov 01 13:05:52.555359 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555342    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tmp-dir" pod "c66adcd6-8708-444a-8781-0d1fee4e7fb8" (UID: "c66adcd6-8708-444a-8781-0d1fee4e7fb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:52.555608 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555350    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-model-cache" pod "c66adcd6-8708-444a-8781-0d1fee4e7fb8" (UID: "c66adcd6-8708-444a-8781-0d1fee4e7fb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:52.555608 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555496    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-home" pod "c66adcd6-8708-444a-8781-0d1fee4e7fb8" (UID: "c66adcd6-8708-444a-8781-0d1fee4e7fb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:52.555716 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555636    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:52.555716 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555654    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:52.555716 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.555668    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:52.557690 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.557664    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66adcd6-8708-444a-8781-0d1fee4e7fb8-kube-api-access-f5bx6" pod "c66adcd6-8708-444a-8781-0d1fee4e7fb8" (UID: "c66adcd6-8708-444a-8781-0d1fee4e7fb8"). InnerVolumeSpecName "kube-api-access-f5bx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:05:52.557867 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.557838    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-dshm" pod "c66adcd6-8708-444a-8781-0d1fee4e7fb8" (UID: "c66adcd6-8708-444a-8781-0d1fee4e7fb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:52.557997 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.557921    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tls-certs" pod "c66adcd6-8708-444a-8781-0d1fee4e7fb8" (UID: "c66adcd6-8708-444a-8781-0d1fee4e7fb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:05:52.656678 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.656649    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66adcd6-8708-444a-8781-0d1fee4e7fb8-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:52.656678 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.656675    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66adcd6-8708-444a-8781-0d1fee4e7fb8-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:52.656867 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.656686    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5bx6\" (UniqueName: \"kubernetes.io/projected/c66adcd6-8708-444a-8781-0d1fee4e7fb8-kube-api-access-f5bx6\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:52.925534 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.925449    5725 generic.go:356] "Generic (PLEG): container finished" podID="c66adcd6-8708-444a-8781-0d1fee4e7fb8" containerID="a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547" exitCode=0
Nov 01 13:05:52.925534 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.925492    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"
Nov 01 13:05:52.925534 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.925502    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc" event={"ID":"c66adcd6-8708-444a-8781-0d1fee4e7fb8","Type":"ContainerDied","Data":"a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547"}
Nov 01 13:05:52.925839 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.925543    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc" event={"ID":"c66adcd6-8708-444a-8781-0d1fee4e7fb8","Type":"ContainerDied","Data":"cf66ed30defc17b1c74233d81c2e0a3d8a795cd384b5578582eb7ba42174afbe"}
Nov 01 13:05:52.925839 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.925566    5725 scope.go:122] "RemoveContainer" containerID="a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547"
Nov 01 13:05:52.935112 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.935092    5725 scope.go:122] "RemoveContainer" containerID="a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547"
Nov 01 13:05:52.935393 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:05:52.935368    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547\": container with ID starting with a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547 not found: ID does not exist" containerID="a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547"
Nov 01 13:05:52.935442 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:52.935402    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547"} err="failed to get container status \"a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547\": rpc error: code = NotFound desc = could not find container \"a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547\": container with ID starting with a7fd574545a7ba16fb8e581c990085634efab38624c97dda27c4ad5461dbd547 not found: ID does not exist"
Nov 01 13:05:53.083159 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:53.083126    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"]
Nov 01 13:05:53.129912 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:53.129875    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b68322/test-model-b68322-kserve-6cf9b7b84f-brwvc"]
Nov 01 13:05:54.615907 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:54.615873    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c66adcd6-8708-444a-8781-0d1fee4e7fb8" path="/var/lib/kubelet/pods/c66adcd6-8708-444a-8781-0d1fee4e7fb8/volumes"
Nov 01 13:05:56.516717 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.516678    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"]
Nov 01 13:05:56.517294 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.517019    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt" podUID="b0ad545d-8951-47b1-887d-8c7f9bfd43cc" containerName="maas-api" containerID="cri-o://4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5" gracePeriod=30
Nov 01 13:05:56.802300 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.802272    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 13:05:56.895152 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.895116    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-openshift-service-ca\") pod \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") "
Nov 01 13:05:56.895367 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.895183    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-kube-api-access-v5jcd\" (UniqueName: \"kubernetes.io/projected/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-kube-api-access-v5jcd\") pod \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") "
Nov 01 13:05:56.895367 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.895212    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-metrics-certs\") pod \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") "
Nov 01 13:05:56.895367 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.895312    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls\") pod \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\" (UID: \"b0ad545d-8951-47b1-887d-8c7f9bfd43cc\") "
Nov 01 13:05:56.895574 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.895543    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-openshift-service-ca" pod "b0ad545d-8951-47b1-887d-8c7f9bfd43cc" (UID: "b0ad545d-8951-47b1-887d-8c7f9bfd43cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:05:56.895720 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.895696    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:56.897425 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.897396    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls" pod "b0ad545d-8951-47b1-887d-8c7f9bfd43cc" (UID: "b0ad545d-8951-47b1-887d-8c7f9bfd43cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:05:56.897550 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.897428    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-metrics-certs" pod "b0ad545d-8951-47b1-887d-8c7f9bfd43cc" (UID: "b0ad545d-8951-47b1-887d-8c7f9bfd43cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:05:56.897550 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.897522    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-kube-api-access-v5jcd" pod "b0ad545d-8951-47b1-887d-8c7f9bfd43cc" (UID: "b0ad545d-8951-47b1-887d-8c7f9bfd43cc"). InnerVolumeSpecName "kube-api-access-v5jcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:05:56.953255 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.953211    5725 generic.go:356] "Generic (PLEG): container finished" podID="b0ad545d-8951-47b1-887d-8c7f9bfd43cc" containerID="4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5" exitCode=0
Nov 01 13:05:56.953451 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.953266    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"
Nov 01 13:05:56.953451 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.953301    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt" event={"ID":"b0ad545d-8951-47b1-887d-8c7f9bfd43cc","Type":"ContainerDied","Data":"4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5"}
Nov 01 13:05:56.953451 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.953358    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt" event={"ID":"b0ad545d-8951-47b1-887d-8c7f9bfd43cc","Type":"ContainerDied","Data":"f23d0f786d84d673287247cfb56a567d96a9d820c96e337030ff6d30ce5a1c26"}
Nov 01 13:05:56.953451 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.953388    5725 scope.go:122] "RemoveContainer" containerID="4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5"
Nov 01 13:05:56.965397 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.965376    5725 scope.go:122] "RemoveContainer" containerID="4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5"
Nov 01 13:05:56.965721 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:05:56.965702    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5\": container with ID starting with 4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5 not found: ID does not exist" containerID="4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5"
Nov 01 13:05:56.965780 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.965731    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5"} err="failed to get container status \"4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5\": rpc error: code = NotFound desc = could not find container \"4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5\": container with ID starting with 4243342c339733eb83a8cdc425f609c9ee587d26d39acf2cbb8259916cdee0a5 not found: ID does not exist"
Nov 01 13:05:56.996217 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.996193    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:56.996217 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.996216    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5jcd\" (UniqueName: \"kubernetes.io/projected/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-kube-api-access-v5jcd\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:56.996396 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:56.996228    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ad545d-8951-47b1-887d-8c7f9bfd43cc-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:57.109322 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:57.109242    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"]
Nov 01 13:05:57.113204 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:57.113176    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-72078a-6949d7b67f-jc7kt"]
Nov 01 13:05:58.615931 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:05:58.615889    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ad545d-8951-47b1-887d-8c7f9bfd43cc" path="/var/lib/kubelet/pods/b0ad545d-8951-47b1-887d-8c7f9bfd43cc/volumes"
Nov 01 13:06:01.026495 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:06:01.026460    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:06:30.696964 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:30.696890    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"]
Nov 01 13:06:30.697509 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:30.697099    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k" podUID="e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" containerName="maas-api" containerID="cri-o://c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1" gracePeriod=30
Nov 01 13:06:30.973285 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:30.973254    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:06:31.026613 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:06:31.026573    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:06:31.048272 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.048244    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls\") pod \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") "
Nov 01 13:06:31.048438 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.048341    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-openshift-service-ca\") pod \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") "
Nov 01 13:06:31.048522 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.048456    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-kube-api-access-9x2s4\" (UniqueName: \"kubernetes.io/projected/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-kube-api-access-9x2s4\") pod \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") "
Nov 01 13:06:31.048522 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.048484    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-metrics-certs\") pod \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\" (UID: \"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d\") "
Nov 01 13:06:31.048764 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.048723    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-openshift-service-ca" pod "e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" (UID: "e8b8f9f6-a438-4a81-9c2a-8de8db1a237d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:06:31.048859 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.048838    5725 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-openshift-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:31.050526 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.050496    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls" pod "e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" (UID: "e8b8f9f6-a438-4a81-9c2a-8de8db1a237d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:06:31.050526 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.050510    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-metrics-certs" pod "e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" (UID: "e8b8f9f6-a438-4a81-9c2a-8de8db1a237d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:06:31.050668 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.050558    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-kube-api-access-9x2s4" pod "e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" (UID: "e8b8f9f6-a438-4a81-9c2a-8de8db1a237d"). InnerVolumeSpecName "kube-api-access-9x2s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:06:31.149787 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.149751    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x2s4\" (UniqueName: \"kubernetes.io/projected/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-kube-api-access-9x2s4\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:31.149787 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.149786    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:31.150011 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.149795    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d-maas-api-tls\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:31.163219 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.163185    5725 generic.go:356] "Generic (PLEG): container finished" podID="e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" containerID="c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1" exitCode=0
Nov 01 13:06:31.163393 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.163240    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"
Nov 01 13:06:31.163393 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.163258    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k" event={"ID":"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d","Type":"ContainerDied","Data":"c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1"}
Nov 01 13:06:31.163393 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.163314    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k" event={"ID":"e8b8f9f6-a438-4a81-9c2a-8de8db1a237d","Type":"ContainerDied","Data":"3f6d4c56203b11b61b5eb23da3b62146bd6f17b11d3f55286af9098d2f3128cd"}
Nov 01 13:06:31.163393 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.163338    5725 scope.go:122] "RemoveContainer" containerID="c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1"
Nov 01 13:06:31.174017 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.173994    5725 scope.go:122] "RemoveContainer" containerID="c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1"
Nov 01 13:06:31.174320 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:06:31.174301    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1\": container with ID starting with c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1 not found: ID does not exist" containerID="c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1"
Nov 01 13:06:31.174389 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.174333    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1"} err="failed to get container status \"c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1\": rpc error: code = NotFound desc = could not find container \"c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1\": container with ID starting with c2124cfce00b9fd849dc97791edb2f15ee90561f6fdd70df110e3169427031f1 not found: ID does not exist"
Nov 01 13:06:31.319354 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.319313    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"]
Nov 01 13:06:31.324203 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:31.324177    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68322-5fb4cfc4c4-s6j4k"]
Nov 01 13:06:32.618339 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:32.618297    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" path="/var/lib/kubelet/pods/e8b8f9f6-a438-4a81-9c2a-8de8db1a237d/volumes"
Nov 01 13:06:40.807888 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.807851    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"]
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808322    5725 cpu_manager.go:414] "containerMap: removing container" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808336    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808346    5725 cpu_manager.go:414] "containerMap: removing container" podUID="d25850a7-529b-41db-8318-054a24043526" containerName="main"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808351    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25850a7-529b-41db-8318-054a24043526" containerName="main"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808359    5725 cpu_manager.go:414] "containerMap: removing container" podUID="2cc6f9cc-7039-4174-967c-d264de6c076e" containerName="payload-pre-processing"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808365    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc6f9cc-7039-4174-967c-d264de6c076e" containerName="payload-pre-processing"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808375    5725 cpu_manager.go:414] "containerMap: removing container" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808380    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808395    5725 cpu_manager.go:414] "containerMap: removing container" podUID="b0ad545d-8951-47b1-887d-8c7f9bfd43cc" containerName="maas-api"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808400    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ad545d-8951-47b1-887d-8c7f9bfd43cc" containerName="maas-api"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808407    5725 cpu_manager.go:414] "containerMap: removing container" podUID="c66adcd6-8708-444a-8781-0d1fee4e7fb8" containerName="main"
Nov 01 13:06:40.808409 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808412    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66adcd6-8708-444a-8781-0d1fee4e7fb8" containerName="main"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808431    5725 cpu_manager.go:414] "containerMap: removing container" podUID="e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" containerName="maas-api"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808436    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" containerName="maas-api"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808517    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b8f9f6-a438-4a81-9c2a-8de8db1a237d" containerName="maas-api"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808527    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8dec32-afbe-4385-a7e9-d57fc37848ac" containerName="main"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808533    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25850a7-529b-41db-8318-054a24043526" containerName="main"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808538    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc6f9cc-7039-4174-967c-d264de6c076e" containerName="payload-pre-processing"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808544    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="77084dd3-4b8d-4314-925d-478db9af9706" containerName="main"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808551    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="385cf49d-a997-42c7-9d2f-52c6dacdd881" containerName="istio-proxy"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808558    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ad545d-8951-47b1-887d-8c7f9bfd43cc" containerName="maas-api"
Nov 01 13:06:40.808828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.808563    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="c66adcd6-8708-444a-8781-0d1fee4e7fb8" containerName="main"
Nov 01 13:06:40.811917 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.811897    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:40.865218 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.865176    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:40.865410 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.865303    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dng8z\" (UniqueName: \"kubernetes.io/projected/7f175605-81bc-4292-b148-11ae56f1f6b9-kube-api-access-dng8z\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:40.865497 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.865474    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:40.908279 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.908243    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:06:40.908472 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.908424    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:06:40.911068 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.911022    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"]
Nov 01 13:06:40.966458 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.966423    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:40.966679 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.966500    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:40.966679 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.966566    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dng8z\" (UniqueName: \"kubernetes.io/projected/7f175605-81bc-4292-b148-11ae56f1f6b9-kube-api-access-dng8z\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:40.967193 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.967169    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:40.969089 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:40.969065    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:41.021281 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:41.021250    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dng8z\" (UniqueName: \"kubernetes.io/projected/7f175605-81bc-4292-b148-11ae56f1f6b9-kube-api-access-dng8z\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:41.125017 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:41.124932    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:41.498868 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:41.498834    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"]
Nov 01 13:06:41.506755 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:06:41.506727    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f175605_81bc_4292_b148_11ae56f1f6b9.slice/crio-9bcd3e5bca11ebb17cae1333caf401be65e42e4c838e239db2ab4dca71921013 WatchSource:0}: Error finding container 9bcd3e5bca11ebb17cae1333caf401be65e42e4c838e239db2ab4dca71921013: Status 404 returned error can't find the container with id 9bcd3e5bca11ebb17cae1333caf401be65e42e4c838e239db2ab4dca71921013
Nov 01 13:06:42.046314 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:42.046273    5725 scope.go:122] "RemoveContainer" containerID="73482b15b0f7beaca1989fbc322d392de476c655316d46acc21139a8211457db"
Nov 01 13:06:42.056221 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:42.056198    5725 scope.go:122] "RemoveContainer" containerID="7be4fcedfc2959832625ab8207ea4d9ef18b1eac1de95d243c534dd629c8c8d6"
Nov 01 13:06:42.240235 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:42.240203    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j" event={"ID":"7f175605-81bc-4292-b148-11ae56f1f6b9","Type":"ContainerStarted","Data":"b92807f2672829a39f0a8543b84c219df75fa046c88f7132fc8b0a84fdd92b55"}
Nov 01 13:06:42.240446 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:42.240251    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j" event={"ID":"7f175605-81bc-4292-b148-11ae56f1f6b9","Type":"ContainerStarted","Data":"9bcd3e5bca11ebb17cae1333caf401be65e42e4c838e239db2ab4dca71921013"}
Nov 01 13:06:42.351149 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:42.351087    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j" podStartSLOduration=2.351066426 podStartE2EDuration="2.351066426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:06:42.347295249 +0000 UTC m=+2643.854486466" watchObservedRunningTime="2026-11-01 13:06:42.351066426 +0000 UTC m=+2643.858257641"
Nov 01 13:06:43.248182 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:43.248146    5725 generic.go:356] "Generic (PLEG): container finished" podID="7f175605-81bc-4292-b148-11ae56f1f6b9" containerID="b92807f2672829a39f0a8543b84c219df75fa046c88f7132fc8b0a84fdd92b55" exitCode=0
Nov 01 13:06:43.248591 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:43.248232    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j" event={"ID":"7f175605-81bc-4292-b148-11ae56f1f6b9","Type":"ContainerDied","Data":"b92807f2672829a39f0a8543b84c219df75fa046c88f7132fc8b0a84fdd92b55"}
Nov 01 13:06:44.406223 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.406196    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:44.504801 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.504766    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f175605-81bc-4292-b148-11ae56f1f6b9-kube-api-access-dng8z\" (UniqueName: \"kubernetes.io/projected/7f175605-81bc-4292-b148-11ae56f1f6b9-kube-api-access-dng8z\") pod \"7f175605-81bc-4292-b148-11ae56f1f6b9\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") "
Nov 01 13:06:44.505009 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.504871    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-xks-ca\") pod \"7f175605-81bc-4292-b148-11ae56f1f6b9\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") "
Nov 01 13:06:44.505009 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.504923    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-openshift-ca\") pod \"7f175605-81bc-4292-b148-11ae56f1f6b9\" (UID: \"7f175605-81bc-4292-b148-11ae56f1f6b9\") "
Nov 01 13:06:44.505284 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.505262    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-openshift-ca" pod "7f175605-81bc-4292-b148-11ae56f1f6b9" (UID: "7f175605-81bc-4292-b148-11ae56f1f6b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:06:44.507270 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.507196    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f175605-81bc-4292-b148-11ae56f1f6b9-kube-api-access-dng8z" pod "7f175605-81bc-4292-b148-11ae56f1f6b9" (UID: "7f175605-81bc-4292-b148-11ae56f1f6b9"). InnerVolumeSpecName "kube-api-access-dng8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:06:44.507270 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.507253    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-xks-ca" pod "7f175605-81bc-4292-b148-11ae56f1f6b9" (UID: "7f175605-81bc-4292-b148-11ae56f1f6b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:06:44.605794 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.605760    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:44.605794 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.605791    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f175605-81bc-4292-b148-11ae56f1f6b9-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:44.605794 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:44.605801    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dng8z\" (UniqueName: \"kubernetes.io/projected/7f175605-81bc-4292-b148-11ae56f1f6b9-kube-api-access-dng8z\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:45.262138 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:45.262112    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"
Nov 01 13:06:45.262334 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:45.262139    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j" event={"ID":"7f175605-81bc-4292-b148-11ae56f1f6b9","Type":"ContainerDied","Data":"9bcd3e5bca11ebb17cae1333caf401be65e42e4c838e239db2ab4dca71921013"}
Nov 01 13:06:45.262334 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:45.262192    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bcd3e5bca11ebb17cae1333caf401be65e42e4c838e239db2ab4dca71921013"
Nov 01 13:06:52.328131 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:52.328096    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"]
Nov 01 13:06:52.442711 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:52.442679    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"]
Nov 01 13:06:54.642374 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.642340    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"]
Nov 01 13:06:54.642831 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.642817    5725 cpu_manager.go:414] "containerMap: removing container" podUID="7f175605-81bc-4292-b148-11ae56f1f6b9" containerName="revoke-keys"
Nov 01 13:06:54.642887 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.642832    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f175605-81bc-4292-b148-11ae56f1f6b9" containerName="revoke-keys"
Nov 01 13:06:54.642926 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.642920    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f175605-81bc-4292-b148-11ae56f1f6b9" containerName="revoke-keys"
Nov 01 13:06:54.646340 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.646322    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:54.698459 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.698427    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"]
Nov 01 13:06:54.741566 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.741531    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:06:54.741841 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.741821    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:06:54.812481 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.812451    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhxk\" (UniqueName: \"kubernetes.io/projected/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-kube-api-access-vdhxk\") pod \"maas-api-revoke-sub-435e608b-gl797\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:54.812649 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.812489    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-435e608b-gl797\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:54.812649 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.812605    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-435e608b-gl797\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:54.913312 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.913228    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhxk\" (UniqueName: \"kubernetes.io/projected/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-kube-api-access-vdhxk\") pod \"maas-api-revoke-sub-435e608b-gl797\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:54.913312 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.913268    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-435e608b-gl797\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:54.913515 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.913329    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-435e608b-gl797\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:54.913955 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.913936    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-435e608b-gl797\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:54.915736 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.915712    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-435e608b-gl797\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:54.967590 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:54.967551    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhxk\" (UniqueName: \"kubernetes.io/projected/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-kube-api-access-vdhxk\") pod \"maas-api-revoke-sub-435e608b-gl797\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:55.259548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:55.259514    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:55.631716 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:55.631635    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"]
Nov 01 13:06:55.639262 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:06:55.639232    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb03a36_ebe2_484b_9a40_9e9b2bc8a042.slice/crio-81babed0b959f9450fb2b5cc0a4d56346d20d067eaa753bd0cf2898acac0cd00 WatchSource:0}: Error finding container 81babed0b959f9450fb2b5cc0a4d56346d20d067eaa753bd0cf2898acac0cd00: Status 404 returned error can't find the container with id 81babed0b959f9450fb2b5cc0a4d56346d20d067eaa753bd0cf2898acac0cd00
Nov 01 13:06:56.336479 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:56.336438    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797" event={"ID":"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042","Type":"ContainerStarted","Data":"d2556966293428c5851c947d782c3a594ccc5efdb82a11a2caa22195e2679306"}
Nov 01 13:06:56.336961 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:56.336506    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797" event={"ID":"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042","Type":"ContainerStarted","Data":"81babed0b959f9450fb2b5cc0a4d56346d20d067eaa753bd0cf2898acac0cd00"}
Nov 01 13:06:56.443957 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:56.443904    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797" podStartSLOduration=2.443892028 podStartE2EDuration="2.443892028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:06:56.442590012 +0000 UTC m=+2657.949781228" watchObservedRunningTime="2026-11-01 13:06:56.443892028 +0000 UTC m=+2657.951083244"
Nov 01 13:06:57.329168 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:57.329095    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr" podUID="e490d69c-9098-4829-9e8d-14e448dc5f74" containerName="payload-processing" containerID="cri-o://7c5c21beb02f31e80ee88104b507ead9b1d4370246a4725f116c1505bad86cf1" gracePeriod=25
Nov 01 13:06:57.343932 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:57.343897    5725 generic.go:356] "Generic (PLEG): container finished" podID="5bb03a36-ebe2-484b-9a40-9e9b2bc8a042" containerID="d2556966293428c5851c947d782c3a594ccc5efdb82a11a2caa22195e2679306" exitCode=0
Nov 01 13:06:57.344333 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:57.343952    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797" event={"ID":"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042","Type":"ContainerDied","Data":"d2556966293428c5851c947d782c3a594ccc5efdb82a11a2caa22195e2679306"}
Nov 01 13:06:57.443395 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:57.443351    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn" podUID="87de366f-a30e-43f0-90ec-9671c04de355" containerName="payload-pre-processing" containerID="cri-o://2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117" gracePeriod=25
Nov 01 13:06:58.497247 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.497221    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:58.557688 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.557652    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-kube-api-access-vdhxk\" (UniqueName: \"kubernetes.io/projected/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-kube-api-access-vdhxk\") pod \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") "
Nov 01 13:06:58.557891 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.557729    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-openshift-ca\") pod \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") "
Nov 01 13:06:58.557891 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.557813    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-xks-ca\") pod \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\" (UID: \"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042\") "
Nov 01 13:06:58.558103 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.558072    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-openshift-ca" pod "5bb03a36-ebe2-484b-9a40-9e9b2bc8a042" (UID: "5bb03a36-ebe2-484b-9a40-9e9b2bc8a042"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:06:58.558194 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.558183    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:58.559897 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.559871    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-xks-ca" pod "5bb03a36-ebe2-484b-9a40-9e9b2bc8a042" (UID: "5bb03a36-ebe2-484b-9a40-9e9b2bc8a042"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:06:58.561144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.561116    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-kube-api-access-vdhxk" pod "5bb03a36-ebe2-484b-9a40-9e9b2bc8a042" (UID: "5bb03a36-ebe2-484b-9a40-9e9b2bc8a042"). InnerVolumeSpecName "kube-api-access-vdhxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:06:58.659127 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.659089    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdhxk\" (UniqueName: \"kubernetes.io/projected/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-kube-api-access-vdhxk\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:58.659127 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:58.659126    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:59.358991 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:59.358952    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"
Nov 01 13:06:59.358991 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:59.358978    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797" event={"ID":"5bb03a36-ebe2-484b-9a40-9e9b2bc8a042","Type":"ContainerDied","Data":"81babed0b959f9450fb2b5cc0a4d56346d20d067eaa753bd0cf2898acac0cd00"}
Nov 01 13:06:59.359274 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:59.359023    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81babed0b959f9450fb2b5cc0a4d56346d20d067eaa753bd0cf2898acac0cd00"
Nov 01 13:06:59.440485 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:59.440446    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn" podUID="87de366f-a30e-43f0-90ec-9671c04de355" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:06:59.440684 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:06:59.440446    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr" podUID="e490d69c-9098-4829-9e8d-14e448dc5f74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:07:01.029587 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:07:01.029546    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:07:07.046429 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:07.046391    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"]
Nov 01 13:07:07.093211 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:07.093180    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7079fd04-2dscr"]
Nov 01 13:07:08.615224 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:08.615193    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be8e5b0-7675-4c26-bf76-aab756c42786" path="/var/lib/kubelet/pods/1be8e5b0-7675-4c26-bf76-aab756c42786/volumes"
Nov 01 13:07:09.440212 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:09.440174    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr" podUID="e490d69c-9098-4829-9e8d-14e448dc5f74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:07:09.440414 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:09.440186    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn" podUID="87de366f-a30e-43f0-90ec-9671c04de355" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 13:07:12.721359 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:12.721319    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"]
Nov 01 13:07:12.768195 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:12.768164    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7427d-gl95j"]
Nov 01 13:07:14.615394 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:14.615357    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f175605-81bc-4292-b148-11ae56f1f6b9" path="/var/lib/kubelet/pods/7f175605-81bc-4292-b148-11ae56f1f6b9/volumes"
Nov 01 13:07:17.475587 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.475558    5725 generic.go:356] "Generic (PLEG): container finished" podID="e490d69c-9098-4829-9e8d-14e448dc5f74" containerID="7c5c21beb02f31e80ee88104b507ead9b1d4370246a4725f116c1505bad86cf1" exitCode=0
Nov 01 13:07:17.475945 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.475627    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr" event={"ID":"e490d69c-9098-4829-9e8d-14e448dc5f74","Type":"ContainerDied","Data":"7c5c21beb02f31e80ee88104b507ead9b1d4370246a4725f116c1505bad86cf1"}
Nov 01 13:07:17.605337 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.605309    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 13:07:17.689280 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.689256    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 13:07:17.758599 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.758531    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87de366f-a30e-43f0-90ec-9671c04de355-kube-api-access-nc2n6\" (UniqueName: \"kubernetes.io/projected/87de366f-a30e-43f0-90ec-9671c04de355-kube-api-access-nc2n6\") pod \"87de366f-a30e-43f0-90ec-9671c04de355\" (UID: \"87de366f-a30e-43f0-90ec-9671c04de355\") "
Nov 01 13:07:17.758599 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.758610    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e490d69c-9098-4829-9e8d-14e448dc5f74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e490d69c-9098-4829-9e8d-14e448dc5f74-plugins-config-volume\") pod \"e490d69c-9098-4829-9e8d-14e448dc5f74\" (UID: \"e490d69c-9098-4829-9e8d-14e448dc5f74\") "
Nov 01 13:07:17.758836 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.758743    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e490d69c-9098-4829-9e8d-14e448dc5f74-kube-api-access-8mgk8\" (UniqueName: \"kubernetes.io/projected/e490d69c-9098-4829-9e8d-14e448dc5f74-kube-api-access-8mgk8\") pod \"e490d69c-9098-4829-9e8d-14e448dc5f74\" (UID: \"e490d69c-9098-4829-9e8d-14e448dc5f74\") "
Nov 01 13:07:17.758836 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.758781    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87de366f-a30e-43f0-90ec-9671c04de355-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de366f-a30e-43f0-90ec-9671c04de355-plugins-config-volume\") pod \"87de366f-a30e-43f0-90ec-9671c04de355\" (UID: \"87de366f-a30e-43f0-90ec-9671c04de355\") "
Nov 01 13:07:17.759211 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.759178    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e490d69c-9098-4829-9e8d-14e448dc5f74-plugins-config-volume" pod "e490d69c-9098-4829-9e8d-14e448dc5f74" (UID: "e490d69c-9098-4829-9e8d-14e448dc5f74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:07:17.759357 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.759277    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87de366f-a30e-43f0-90ec-9671c04de355-plugins-config-volume" pod "87de366f-a30e-43f0-90ec-9671c04de355" (UID: "87de366f-a30e-43f0-90ec-9671c04de355"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:07:17.760946 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.760912    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87de366f-a30e-43f0-90ec-9671c04de355-kube-api-access-nc2n6" pod "87de366f-a30e-43f0-90ec-9671c04de355" (UID: "87de366f-a30e-43f0-90ec-9671c04de355"). InnerVolumeSpecName "kube-api-access-nc2n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:07:17.761084 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.760964    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e490d69c-9098-4829-9e8d-14e448dc5f74-kube-api-access-8mgk8" pod "e490d69c-9098-4829-9e8d-14e448dc5f74" (UID: "e490d69c-9098-4829-9e8d-14e448dc5f74"). InnerVolumeSpecName "kube-api-access-8mgk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:07:17.860107 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.860008    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mgk8\" (UniqueName: \"kubernetes.io/projected/e490d69c-9098-4829-9e8d-14e448dc5f74-kube-api-access-8mgk8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:17.860107 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.860038    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87de366f-a30e-43f0-90ec-9671c04de355-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:17.860107 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.860068    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc2n6\" (UniqueName: \"kubernetes.io/projected/87de366f-a30e-43f0-90ec-9671c04de355-kube-api-access-nc2n6\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:17.860107 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:17.860078    5725 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e490d69c-9098-4829-9e8d-14e448dc5f74-plugins-config-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:18.482371 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.482323    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr" event={"ID":"e490d69c-9098-4829-9e8d-14e448dc5f74","Type":"ContainerDied","Data":"8320dfa9bbaaf59cc58a61776dcf30c3c0307a7e653297412f1cd3c29a718dba"}
Nov 01 13:07:18.482868 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.482345    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"
Nov 01 13:07:18.482868 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.482424    5725 scope.go:122] "RemoveContainer" containerID="7c5c21beb02f31e80ee88104b507ead9b1d4370246a4725f116c1505bad86cf1"
Nov 01 13:07:18.483944 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.483763    5725 generic.go:356] "Generic (PLEG): container finished" podID="87de366f-a30e-43f0-90ec-9671c04de355" containerID="2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117" exitCode=0
Nov 01 13:07:18.483944 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.483816    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"
Nov 01 13:07:18.483944 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.483837    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn" event={"ID":"87de366f-a30e-43f0-90ec-9671c04de355","Type":"ContainerDied","Data":"2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117"}
Nov 01 13:07:18.483944 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.483889    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn" event={"ID":"87de366f-a30e-43f0-90ec-9671c04de355","Type":"ContainerDied","Data":"c35f9c89f9fed2fc30dfe5f8e061f6959b10f2a4fdfcdc8cd1e5ae92474a338f"}
Nov 01 13:07:18.494695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.494677    5725 scope.go:122] "RemoveContainer" containerID="2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117"
Nov 01 13:07:18.507723 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.507695    5725 scope.go:122] "RemoveContainer" containerID="2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117"
Nov 01 13:07:18.508191 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:07:18.508163    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117\": container with ID starting with 2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117 not found: ID does not exist" containerID="2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117"
Nov 01 13:07:18.508310 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.508204    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117"} err="failed to get container status \"2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117\": rpc error: code = NotFound desc = could not find container \"2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117\": container with ID starting with 2e00b38bcb41363e58494f86176f1617d90a967d02f5be28b0180aaecd0a0117 not found: ID does not exist"
Nov 01 13:07:18.636943 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.636865    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"]
Nov 01 13:07:18.683701 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.683667    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f7427d-b4c994959-x4wcr"]
Nov 01 13:07:18.748844 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.748811    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"]
Nov 01 13:07:18.752579 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:18.752552    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f7427d-74d8ff968c-b25fn"]
Nov 01 13:07:19.388828 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:19.388787    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"]
Nov 01 13:07:19.389109 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:19.388987    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy" containerID="cri-o://f13cfc0ba614cc28466330be96ab29221b11f8d32a0e9173f0ed932de8831e72" gracePeriod=30
Nov 01 13:07:20.615826 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:20.615797    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87de366f-a30e-43f0-90ec-9671c04de355" path="/var/lib/kubelet/pods/87de366f-a30e-43f0-90ec-9671c04de355/volumes"
Nov 01 13:07:20.616247 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:20.616212    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e490d69c-9098-4829-9e8d-14e448dc5f74" path="/var/lib/kubelet/pods/e490d69c-9098-4829-9e8d-14e448dc5f74/volumes"
Nov 01 13:07:22.300080 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:22.300014    5725 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 13:07:22.300586 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:22.300079    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 13:07:24.554957 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.554917    5725 generic.go:356] "Generic (PLEG): container finished" podID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerID="f13cfc0ba614cc28466330be96ab29221b11f8d32a0e9173f0ed932de8831e72" exitCode=0
Nov 01 13:07:24.555373 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.554998    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" event={"ID":"dc384213-bc6e-4d61-943a-5e88e4eb1ea4","Type":"ContainerDied","Data":"f13cfc0ba614cc28466330be96ab29221b11f8d32a0e9173f0ed932de8831e72"}
Nov 01 13:07:24.661140 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.661077    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 13:07:24.674499 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.674473    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57458d666c-mbkth"]
Nov 01 13:07:24.674692 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.674669    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-57458d666c-mbkth" podUID="a52ae595-903b-4c45-9802-e106d8519f07" containerName="manager" containerID="cri-o://5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3" gracePeriod=30
Nov 01 13:07:24.832554 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832504    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-certs\") pod \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") "
Nov 01 13:07:24.832752 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832568    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-kube-api-access-j5ghv\") pod \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") "
Nov 01 13:07:24.832752 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832603    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-credential-socket\") pod \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") "
Nov 01 13:07:24.832752 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832645    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-podinfo\") pod \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") "
Nov 01 13:07:24.832752 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832693    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-socket\") pod \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") "
Nov 01 13:07:24.832752 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832745    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-data\") pod \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") "
Nov 01 13:07:24.833483 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832785    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-envoy\") pod \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") "
Nov 01 13:07:24.833483 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832800    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-certs" pod "dc384213-bc6e-4d61-943a-5e88e4eb1ea4" (UID: "dc384213-bc6e-4d61-943a-5e88e4eb1ea4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:07:24.833483 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832849    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-token\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-token\") pod \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") "
Nov 01 13:07:24.833483 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.832921    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istiod-ca-cert\") pod \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\" (UID: \"dc384213-bc6e-4d61-943a-5e88e4eb1ea4\") "
Nov 01 13:07:24.833483 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.833412    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-data" pod "dc384213-bc6e-4d61-943a-5e88e4eb1ea4" (UID: "dc384213-bc6e-4d61-943a-5e88e4eb1ea4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:07:24.833799 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.833615    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-credential-socket" pod "dc384213-bc6e-4d61-943a-5e88e4eb1ea4" (UID: "dc384213-bc6e-4d61-943a-5e88e4eb1ea4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:07:24.834090 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.833967    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-socket" pod "dc384213-bc6e-4d61-943a-5e88e4eb1ea4" (UID: "dc384213-bc6e-4d61-943a-5e88e4eb1ea4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:07:24.834090 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.834019    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istiod-ca-cert" pod "dc384213-bc6e-4d61-943a-5e88e4eb1ea4" (UID: "dc384213-bc6e-4d61-943a-5e88e4eb1ea4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:07:24.834512 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.834473    5725 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:24.834512 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.834500    5725 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-data\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:24.834693 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.834514    5725 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istiod-ca-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:24.834693 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.834528    5725 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-workload-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:24.834693 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.834542    5725 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-credential-socket\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:24.836105 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.836082    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-kube-api-access-j5ghv" pod "dc384213-bc6e-4d61-943a-5e88e4eb1ea4" (UID: "dc384213-bc6e-4d61-943a-5e88e4eb1ea4"). InnerVolumeSpecName "kube-api-access-j5ghv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:07:24.836672 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.836649    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-envoy" pod "dc384213-bc6e-4d61-943a-5e88e4eb1ea4" (UID: "dc384213-bc6e-4d61-943a-5e88e4eb1ea4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:07:24.837625 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.837598    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-token" pod "dc384213-bc6e-4d61-943a-5e88e4eb1ea4" (UID: "dc384213-bc6e-4d61-943a-5e88e4eb1ea4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:07:24.839071 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.839023    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-podinfo" pod "dc384213-bc6e-4d61-943a-5e88e4eb1ea4" (UID: "dc384213-bc6e-4d61-943a-5e88e4eb1ea4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:07:24.936196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.936112    5725 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-token\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:24.936196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.936153    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5ghv\" (UniqueName: \"kubernetes.io/projected/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-kube-api-access-j5ghv\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:24.936196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.936171    5725 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-podinfo\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:24.936196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:24.936185    5725 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc384213-bc6e-4d61-943a-5e88e4eb1ea4-istio-envoy\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:25.025155 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.025129    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 13:07:25.138565 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.138530    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-metrics-certs\") pod \"a52ae595-903b-4c45-9802-e106d8519f07\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") "
Nov 01 13:07:25.138765 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.138573    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52ae595-903b-4c45-9802-e106d8519f07-kube-api-access-bxfxm\" (UniqueName: \"kubernetes.io/projected/a52ae595-903b-4c45-9802-e106d8519f07-kube-api-access-bxfxm\") pod \"a52ae595-903b-4c45-9802-e106d8519f07\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") "
Nov 01 13:07:25.138765 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.138722    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-webhook-cert\") pod \"a52ae595-903b-4c45-9802-e106d8519f07\" (UID: \"a52ae595-903b-4c45-9802-e106d8519f07\") "
Nov 01 13:07:25.144267 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.144223    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52ae595-903b-4c45-9802-e106d8519f07-kube-api-access-bxfxm" pod "a52ae595-903b-4c45-9802-e106d8519f07" (UID: "a52ae595-903b-4c45-9802-e106d8519f07"). InnerVolumeSpecName "kube-api-access-bxfxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:07:25.144488 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.144299    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-metrics-certs" pod "a52ae595-903b-4c45-9802-e106d8519f07" (UID: "a52ae595-903b-4c45-9802-e106d8519f07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:07:25.144488 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.144345    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-webhook-cert" pod "a52ae595-903b-4c45-9802-e106d8519f07" (UID: "a52ae595-903b-4c45-9802-e106d8519f07"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:07:25.240222 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.240185    5725 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-metrics-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:25.240222 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.240219    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxfxm\" (UniqueName: \"kubernetes.io/projected/a52ae595-903b-4c45-9802-e106d8519f07-kube-api-access-bxfxm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:25.240480 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.240234    5725 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a52ae595-903b-4c45-9802-e106d8519f07-webhook-cert\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:25.563712 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.563615    5725 generic.go:356] "Generic (PLEG): container finished" podID="a52ae595-903b-4c45-9802-e106d8519f07" containerID="5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3" exitCode=0
Nov 01 13:07:25.563712 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.563700    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57458d666c-mbkth"
Nov 01 13:07:25.564294 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.563700    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mbkth" event={"ID":"a52ae595-903b-4c45-9802-e106d8519f07","Type":"ContainerDied","Data":"5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3"}
Nov 01 13:07:25.564294 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.563828    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57458d666c-mbkth" event={"ID":"a52ae595-903b-4c45-9802-e106d8519f07","Type":"ContainerDied","Data":"9b5c33da725a9439d39c8bba8827a0b895e3795d70a6913056900a6ecd010288"}
Nov 01 13:07:25.564294 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.563862    5725 scope.go:122] "RemoveContainer" containerID="5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3"
Nov 01 13:07:25.565554 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.565522    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld" event={"ID":"dc384213-bc6e-4d61-943a-5e88e4eb1ea4","Type":"ContainerDied","Data":"1c383a28d3ba490eee66586f778f438f893ea2fa743e4664b7012c0083719d95"}
Nov 01 13:07:25.565674 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.565571    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"
Nov 01 13:07:25.575075 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.574897    5725 scope.go:122] "RemoveContainer" containerID="72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38"
Nov 01 13:07:25.584229 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.584211    5725 scope.go:122] "RemoveContainer" containerID="5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3"
Nov 01 13:07:25.584494 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:07:25.584476    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3\": container with ID starting with 5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3 not found: ID does not exist" containerID="5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3"
Nov 01 13:07:25.584548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.584503    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3"} err="failed to get container status \"5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3\": rpc error: code = NotFound desc = could not find container \"5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3\": container with ID starting with 5b3a0926bbc4fc82bcf88851126fe88c7092d941ef6a302e2ef562a814adebd3 not found: ID does not exist"
Nov 01 13:07:25.584548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.584520    5725 scope.go:122] "RemoveContainer" containerID="72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38"
Nov 01 13:07:25.584757 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:07:25.584735    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38\": container with ID starting with 72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38 not found: ID does not exist" containerID="72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38"
Nov 01 13:07:25.584830 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.584765    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38"} err="failed to get container status \"72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38\": rpc error: code = NotFound desc = could not find container \"72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38\": container with ID starting with 72d497fd969d921334f8ce17ab0fb051c1e518da903d8afa35353f298a371e38 not found: ID does not exist"
Nov 01 13:07:25.584830 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.584785    5725 scope.go:122] "RemoveContainer" containerID="f13cfc0ba614cc28466330be96ab29221b11f8d32a0e9173f0ed932de8831e72"
Nov 01 13:07:25.719240 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.719209    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57458d666c-mbkth"]
Nov 01 13:07:25.766123 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.766084    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57458d666c-mbkth"]
Nov 01 13:07:25.869737 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.869653    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"]
Nov 01 13:07:25.872752 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:25.872723    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-k8bld"]
Nov 01 13:07:26.615167 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:26.615133    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52ae595-903b-4c45-9802-e106d8519f07" path="/var/lib/kubelet/pods/a52ae595-903b-4c45-9802-e106d8519f07/volumes"
Nov 01 13:07:26.615595 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:26.615582    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" path="/var/lib/kubelet/pods/dc384213-bc6e-4d61-943a-5e88e4eb1ea4/volumes"
Nov 01 13:07:31.026308 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:07:31.026272    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:07:39.208170 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:39.208140    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:07:39.215894 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:39.215870    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:07:39.222128 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:39.222098    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:07:39.228860 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:07:39.228841    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:08:01.026212 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:08:01.026120    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:08:31.026182 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:08:31.026141    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:08:42.132175 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:08:42.132141    5725 scope.go:122] "RemoveContainer" containerID="8ea6dc50ea559f1bda933f5bdcfde7afd56ff42519ef31b6519dbdc9c4abb89c"
Nov 01 13:09:01.026140 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:09:01.026099    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:09:31.027969 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:09:31.027887    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:10:01.026418 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:10:01.026372    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:10:31.029763 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:10:31.029720    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:11:01.027180 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:11:01.027100    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:11:21.135252 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:11:21.135202    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 13:11:21.136025 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:11:21.135971    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:11:31.027017 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:11:31.026977    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:11:50.361198 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:11:50.361156    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rxn6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 13:11:50.361851 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:11:50.361814    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rxn6r" podUID="63317aed-7a2c-41fe-8eee-ece29d7326cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:11:59.051011 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:11:59.050970    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"]
Nov 01 13:11:59.097433 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:11:59.097403    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-435e608b-gl797"]
Nov 01 13:12:00.615411 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:12:00.615376    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb03a36-ebe2-484b-9a40-9e9b2bc8a042" path="/var/lib/kubelet/pods/5bb03a36-ebe2-484b-9a40-9e9b2bc8a042/volumes"
Nov 01 13:12:01.026396 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:12:01.026362    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:12:31.026442 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:12:31.026363    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:12:39.260734 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:12:39.260697    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:12:39.268716 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:12:39.268683    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:12:39.272779 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:12:39.272754    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:12:39.280597 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:12:39.280574    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:12:42.180202 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:12:42.180168    5725 scope.go:122] "RemoveContainer" containerID="b92807f2672829a39f0a8543b84c219df75fa046c88f7132fc8b0a84fdd92b55"
Nov 01 13:12:51.143242 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:12:51.143202    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 13:12:51.143894 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:12:51.143860    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:13:01.026279 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:13:01.026238    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:13:31.026331 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:13:31.026286    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:13:42.202526 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:13:42.202490    5725 scope.go:122] "RemoveContainer" containerID="d2556966293428c5851c947d782c3a594ccc5efdb82a11a2caa22195e2679306"
Nov 01 13:14:01.026626 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:14:01.026514    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:14:31.026646 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:14:31.026603    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:15:01.026208 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:15:01.026168    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:15:26.616203 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616133    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"]
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616487    5725 cpu_manager.go:414] "containerMap: removing container" podUID="5bb03a36-ebe2-484b-9a40-9e9b2bc8a042" containerName="revoke-keys"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616499    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb03a36-ebe2-484b-9a40-9e9b2bc8a042" containerName="revoke-keys"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616517    5725 cpu_manager.go:414] "containerMap: removing container" podUID="e490d69c-9098-4829-9e8d-14e448dc5f74" containerName="payload-processing"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616522    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e490d69c-9098-4829-9e8d-14e448dc5f74" containerName="payload-processing"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616531    5725 cpu_manager.go:414] "containerMap: removing container" podUID="a52ae595-903b-4c45-9802-e106d8519f07" containerName="manager"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616536    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52ae595-903b-4c45-9802-e106d8519f07" containerName="manager"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616549    5725 cpu_manager.go:414] "containerMap: removing container" podUID="87de366f-a30e-43f0-90ec-9671c04de355" containerName="payload-pre-processing"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616555    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87de366f-a30e-43f0-90ec-9671c04de355" containerName="payload-pre-processing"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616567    5725 cpu_manager.go:414] "containerMap: removing container" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616572    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616633    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc384213-bc6e-4d61-943a-5e88e4eb1ea4" containerName="istio-proxy"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616643    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52ae595-903b-4c45-9802-e106d8519f07" containerName="manager"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616650    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb03a36-ebe2-484b-9a40-9e9b2bc8a042" containerName="revoke-keys"
Nov 01 13:15:26.616654 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616659    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="e490d69c-9098-4829-9e8d-14e448dc5f74" containerName="payload-processing"
Nov 01 13:15:26.617116 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.616667    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="87de366f-a30e-43f0-90ec-9671c04de355" containerName="payload-pre-processing"
Nov 01 13:15:26.619584 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.619567    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.665117 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.665077    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8c9\" (UniqueName: \"kubernetes.io/projected/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-kube-api-access-2j8c9\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.665298 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.665189    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-home\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.665354 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.665301    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tls-certs\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.665354 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.665336    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-model-cache\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.665441 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.665395    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tmp-dir\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.665441 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.665421    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-dshm\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.693244 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.693213    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"]
Nov 01 13:15:26.717697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.717662    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-483de8\"/\"openshift-service-ca.crt\""
Nov 01 13:15:26.717697 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.717684    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-483de8\"/\"kube-root-ca.crt\""
Nov 01 13:15:26.717878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.717740    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-483de8\"/\"default-dockercfg-lzd9x\""
Nov 01 13:15:26.717878 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.717804    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-483de8\"/\"simulator-single-main-483de8-kserve-self-signed-certs\""
Nov 01 13:15:26.766477 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.766395    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-dshm\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.766953 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.766840    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8c9\" (UniqueName: \"kubernetes.io/projected/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-kube-api-access-2j8c9\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.766953 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.766905    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-home\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.767314 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.767274    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tls-certs\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.767552 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.767449    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-model-cache\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.767552 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.767509    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tmp-dir\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.768536 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.768418    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tmp-dir\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.768752 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.768694    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-model-cache\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.768994 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.768974    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-home\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.771076 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.771041    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-dshm\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.771998 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.771975    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tls-certs\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.822364 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.822325    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8c9\" (UniqueName: \"kubernetes.io/projected/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-kube-api-access-2j8c9\") pod \"simulator-single-main-483de8-kserve-5f6bd974dd-87gkd\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") " pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:26.931723 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:26.931644    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:27.303880 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:27.303853    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"]
Nov 01 13:15:27.305384 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:15:27.305359    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7002dfef_a0c0_4c91_b46e_d1d14d9045fc.slice/crio-c651d37a418944869d61f8f84c81ffa11e1f2ac2b0ccf833c76e1512f75d8408 WatchSource:0}: Error finding container c651d37a418944869d61f8f84c81ffa11e1f2ac2b0ccf833c76e1512f75d8408: Status 404 returned error can't find the container with id c651d37a418944869d61f8f84c81ffa11e1f2ac2b0ccf833c76e1512f75d8408
Nov 01 13:15:27.602758 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:27.602654    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd" event={"ID":"7002dfef-a0c0-4c91-b46e-d1d14d9045fc","Type":"ContainerStarted","Data":"e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528"}
Nov 01 13:15:27.602758 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:27.602705    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd" event={"ID":"7002dfef-a0c0-4c91-b46e-d1d14d9045fc","Type":"ContainerStarted","Data":"c651d37a418944869d61f8f84c81ffa11e1f2ac2b0ccf833c76e1512f75d8408"}
Nov 01 13:15:27.712077 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:27.712000    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd" podStartSLOduration=1.711985298 podStartE2EDuration="1.711985298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:15:27.710130692 +0000 UTC m=+3169.217321911" watchObservedRunningTime="2026-11-01 13:15:27.711985298 +0000 UTC m=+3169.219176514"
Nov 01 13:15:27.932422 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:27.932329    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:31.029011 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:15:31.028959    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:15:36.932292 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:36.932258    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:36.945339 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:36.945314    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:15:37.677886 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:15:37.677859    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:16:01.027755 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:16:01.027706    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:16:28.048019 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:28.047981    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"]
Nov 01 13:16:31.027860 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:16:31.027813    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:16:43.067825 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.067758    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd" podUID="7002dfef-a0c0-4c91-b46e-d1d14d9045fc" containerName="main" containerID="cri-o://e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528" gracePeriod=45
Nov 01 13:16:43.324116 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.324037    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:16:43.413000 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.412961    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tls-certs\") pod \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") "
Nov 01 13:16:43.413202 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.413006    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-dshm\") pod \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") "
Nov 01 13:16:43.413202 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.413034    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-kube-api-access-2j8c9\" (UniqueName: \"kubernetes.io/projected/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-kube-api-access-2j8c9\") pod \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") "
Nov 01 13:16:43.413202 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.413111    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tmp-dir\") pod \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") "
Nov 01 13:16:43.413202 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.413137    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-model-cache\") pod \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") "
Nov 01 13:16:43.413202 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.413170    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-home\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-home\") pod \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\" (UID: \"7002dfef-a0c0-4c91-b46e-d1d14d9045fc\") "
Nov 01 13:16:43.413494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.413447    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tmp-dir" pod "7002dfef-a0c0-4c91-b46e-d1d14d9045fc" (UID: "7002dfef-a0c0-4c91-b46e-d1d14d9045fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:16:43.413494 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.413465    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-model-cache" pod "7002dfef-a0c0-4c91-b46e-d1d14d9045fc" (UID: "7002dfef-a0c0-4c91-b46e-d1d14d9045fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:16:43.413584 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.413474    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-home" pod "7002dfef-a0c0-4c91-b46e-d1d14d9045fc" (UID: "7002dfef-a0c0-4c91-b46e-d1d14d9045fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:16:43.415342 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.415320    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-dshm" pod "7002dfef-a0c0-4c91-b46e-d1d14d9045fc" (UID: "7002dfef-a0c0-4c91-b46e-d1d14d9045fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:16:43.415432 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.415369    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tls-certs" pod "7002dfef-a0c0-4c91-b46e-d1d14d9045fc" (UID: "7002dfef-a0c0-4c91-b46e-d1d14d9045fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:16:43.415471 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.415427    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-kube-api-access-2j8c9" pod "7002dfef-a0c0-4c91-b46e-d1d14d9045fc" (UID: "7002dfef-a0c0-4c91-b46e-d1d14d9045fc"). InnerVolumeSpecName "kube-api-access-2j8c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:16:43.514281 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.514242    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:43.514281 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.514283    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:43.514555 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.514297    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j8c9\" (UniqueName: \"kubernetes.io/projected/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-kube-api-access-2j8c9\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:43.514555 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.514312    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:43.514555 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.514326    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:43.514555 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:43.514339    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7002dfef-a0c0-4c91-b46e-d1d14d9045fc-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:44.051274 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.051241    5725 generic.go:356] "Generic (PLEG): container finished" podID="7002dfef-a0c0-4c91-b46e-d1d14d9045fc" containerID="e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528" exitCode=0
Nov 01 13:16:44.051469 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.051286    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"
Nov 01 13:16:44.051469 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.051324    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd" event={"ID":"7002dfef-a0c0-4c91-b46e-d1d14d9045fc","Type":"ContainerDied","Data":"e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528"}
Nov 01 13:16:44.051469 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.051370    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd" event={"ID":"7002dfef-a0c0-4c91-b46e-d1d14d9045fc","Type":"ContainerDied","Data":"c651d37a418944869d61f8f84c81ffa11e1f2ac2b0ccf833c76e1512f75d8408"}
Nov 01 13:16:44.051469 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.051390    5725 scope.go:122] "RemoveContainer" containerID="e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528"
Nov 01 13:16:44.060933 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.060910    5725 scope.go:122] "RemoveContainer" containerID="e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528"
Nov 01 13:16:44.061216 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:16:44.061198    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528\": container with ID starting with e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528 not found: ID does not exist" containerID="e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528"
Nov 01 13:16:44.061277 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.061226    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528"} err="failed to get container status \"e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528\": rpc error: code = NotFound desc = could not find container \"e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528\": container with ID starting with e1567403473fe7e47c2211f97c63e95e28adbab7d65bffde9925173768976528 not found: ID does not exist"
Nov 01 13:16:44.205502 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.205462    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"]
Nov 01 13:16:44.252669 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.252636    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-483de8/simulator-single-main-483de8-kserve-5f6bd974dd-87gkd"]
Nov 01 13:16:44.614727 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:16:44.614692    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7002dfef-a0c0-4c91-b46e-d1d14d9045fc" path="/var/lib/kubelet/pods/7002dfef-a0c0-4c91-b46e-d1d14d9045fc/volumes"
Nov 01 13:17:01.026750 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:17:01.026665    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:17:21.611059 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:21.611012    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"]
Nov 01 13:17:21.611511 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:21.611176    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t" podUID="96289b27-9274-4f9f-9023-3173737f12e3" containerName="manager" containerID="cri-o://16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0" gracePeriod=10
Nov 01 13:17:21.877415 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:21.877348    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 13:17:21.989367 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:21.989331    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96289b27-9274-4f9f-9023-3173737f12e3-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96289b27-9274-4f9f-9023-3173737f12e3-extensions-socket-volume\") pod \"96289b27-9274-4f9f-9023-3173737f12e3\" (UID: \"96289b27-9274-4f9f-9023-3173737f12e3\") "
Nov 01 13:17:21.989622 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:21.989425    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96289b27-9274-4f9f-9023-3173737f12e3-kube-api-access-pqq8f\" (UniqueName: \"kubernetes.io/projected/96289b27-9274-4f9f-9023-3173737f12e3-kube-api-access-pqq8f\") pod \"96289b27-9274-4f9f-9023-3173737f12e3\" (UID: \"96289b27-9274-4f9f-9023-3173737f12e3\") "
Nov 01 13:17:21.989855 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:21.989805    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96289b27-9274-4f9f-9023-3173737f12e3-extensions-socket-volume" pod "96289b27-9274-4f9f-9023-3173737f12e3" (UID: "96289b27-9274-4f9f-9023-3173737f12e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:17:21.991691 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:21.991665    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96289b27-9274-4f9f-9023-3173737f12e3-kube-api-access-pqq8f" pod "96289b27-9274-4f9f-9023-3173737f12e3" (UID: "96289b27-9274-4f9f-9023-3173737f12e3"). InnerVolumeSpecName "kube-api-access-pqq8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:17:22.091031 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.090993    5725 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96289b27-9274-4f9f-9023-3173737f12e3-extensions-socket-volume\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:17:22.091031 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.091024    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqq8f\" (UniqueName: \"kubernetes.io/projected/96289b27-9274-4f9f-9023-3173737f12e3-kube-api-access-pqq8f\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:17:22.283066 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.283021    5725 generic.go:356] "Generic (PLEG): container finished" podID="96289b27-9274-4f9f-9023-3173737f12e3" containerID="16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0" exitCode=0
Nov 01 13:17:22.283260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.283083    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t" event={"ID":"96289b27-9274-4f9f-9023-3173737f12e3","Type":"ContainerDied","Data":"16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0"}
Nov 01 13:17:22.283260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.283136    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t" event={"ID":"96289b27-9274-4f9f-9023-3173737f12e3","Type":"ContainerDied","Data":"0b140c6e37c52e57ecfcd9dd2b063d4abe03e2c7c013300d83603b076a6359cf"}
Nov 01 13:17:22.283260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.283158    5725 scope.go:122] "RemoveContainer" containerID="16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0"
Nov 01 13:17:22.283260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.283094    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"
Nov 01 13:17:22.297461 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.297442    5725 scope.go:122] "RemoveContainer" containerID="16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0"
Nov 01 13:17:22.297747 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:17:22.297725    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0\": container with ID starting with 16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0 not found: ID does not exist" containerID="16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0"
Nov 01 13:17:22.297811 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.297758    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0"} err="failed to get container status \"16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0\": rpc error: code = NotFound desc = could not find container \"16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0\": container with ID starting with 16e93ce750d2f42b92d7f802f2a2de7172cc85cdfa5b4176e3c7b720549e6ea0 not found: ID does not exist"
Nov 01 13:17:22.438185 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.438149    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"]
Nov 01 13:17:22.485091 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.485059    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tj76t"]
Nov 01 13:17:22.615519 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:22.615436    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96289b27-9274-4f9f-9023-3173737f12e3" path="/var/lib/kubelet/pods/96289b27-9274-4f9f-9023-3173737f12e3/volumes"
Nov 01 13:17:31.026950 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:17:31.026912    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:17:39.301101 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:39.301071    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:17:39.308662 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:39.308638    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:17:39.316081 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:39.316044    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:17:39.323564 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:17:39.323544    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:18:01.026602 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:18:01.026565    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:18:25.199210 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199130    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"]
Nov 01 13:18:25.199695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199555    5725 cpu_manager.go:414] "containerMap: removing container" podUID="7002dfef-a0c0-4c91-b46e-d1d14d9045fc" containerName="main"
Nov 01 13:18:25.199695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199566    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7002dfef-a0c0-4c91-b46e-d1d14d9045fc" containerName="main"
Nov 01 13:18:25.199695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199580    5725 cpu_manager.go:414] "containerMap: removing container" podUID="a52ae595-903b-4c45-9802-e106d8519f07" containerName="manager"
Nov 01 13:18:25.199695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199585    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52ae595-903b-4c45-9802-e106d8519f07" containerName="manager"
Nov 01 13:18:25.199695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199595    5725 cpu_manager.go:414] "containerMap: removing container" podUID="96289b27-9274-4f9f-9023-3173737f12e3" containerName="manager"
Nov 01 13:18:25.199695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199601    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96289b27-9274-4f9f-9023-3173737f12e3" containerName="manager"
Nov 01 13:18:25.199695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199665    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="7002dfef-a0c0-4c91-b46e-d1d14d9045fc" containerName="main"
Nov 01 13:18:25.199695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199676    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="96289b27-9274-4f9f-9023-3173737f12e3" containerName="manager"
Nov 01 13:18:25.199695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.199684    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52ae595-903b-4c45-9802-e106d8519f07" containerName="manager"
Nov 01 13:18:25.202763 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.202747    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:25.246889 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.246861    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"]
Nov 01 13:18:25.270609 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.270582    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/838325b1-57bd-4388-a34f-e524c094fcc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bx6xs\" (UID: \"838325b1-57bd-4388-a34f-e524c094fcc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:25.270609 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.270611    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvvtc\" (UniqueName: \"kubernetes.io/projected/838325b1-57bd-4388-a34f-e524c094fcc1-kube-api-access-tvvtc\") pod \"kuadrant-operator-controller-manager-678d87766f-bx6xs\" (UID: \"838325b1-57bd-4388-a34f-e524c094fcc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:25.291301 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.291274    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xwdxs\""
Nov 01 13:18:25.371929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.371892    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/838325b1-57bd-4388-a34f-e524c094fcc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bx6xs\" (UID: \"838325b1-57bd-4388-a34f-e524c094fcc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:25.371929 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.371931    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvvtc\" (UniqueName: \"kubernetes.io/projected/838325b1-57bd-4388-a34f-e524c094fcc1-kube-api-access-tvvtc\") pod \"kuadrant-operator-controller-manager-678d87766f-bx6xs\" (UID: \"838325b1-57bd-4388-a34f-e524c094fcc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:25.372305 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.372281    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/838325b1-57bd-4388-a34f-e524c094fcc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bx6xs\" (UID: \"838325b1-57bd-4388-a34f-e524c094fcc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:25.426543 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.426515    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvvtc\" (UniqueName: \"kubernetes.io/projected/838325b1-57bd-4388-a34f-e524c094fcc1-kube-api-access-tvvtc\") pod \"kuadrant-operator-controller-manager-678d87766f-bx6xs\" (UID: \"838325b1-57bd-4388-a34f-e524c094fcc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:25.515507 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.515472    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:25.943389 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:25.943362    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"]
Nov 01 13:18:25.945059 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:18:25.945021    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838325b1_57bd_4388_a34f_e524c094fcc1.slice/crio-520db4dd2332e73bc3612d43dcff44047c6ecf536f5b6882af707f57b86a1141 WatchSource:0}: Error finding container 520db4dd2332e73bc3612d43dcff44047c6ecf536f5b6882af707f57b86a1141: Status 404 returned error can't find the container with id 520db4dd2332e73bc3612d43dcff44047c6ecf536f5b6882af707f57b86a1141
Nov 01 13:18:26.631386 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:26.631346    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs" event={"ID":"838325b1-57bd-4388-a34f-e524c094fcc1","Type":"ContainerStarted","Data":"8e9a4c4a9fc96fe252d6d614fb4fcfd7e42a95b27efc7c5cecdd1628f4e8a7e8"}
Nov 01 13:18:26.631881 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:26.631406    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs" event={"ID":"838325b1-57bd-4388-a34f-e524c094fcc1","Type":"ContainerStarted","Data":"520db4dd2332e73bc3612d43dcff44047c6ecf536f5b6882af707f57b86a1141"}
Nov 01 13:18:26.631881 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:26.631433    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:26.853176 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:26.853123    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs" podStartSLOduration=1.853110385 podStartE2EDuration="1.853110385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:18:26.805289193 +0000 UTC m=+3348.312480409" watchObservedRunningTime="2026-11-01 13:18:26.853110385 +0000 UTC m=+3348.360301601"
Nov 01 13:18:31.027143 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:18:31.027104    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:18:37.639721 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:37.639685    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bx6xs"
Nov 01 13:18:49.436219 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:49.436179    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ns4rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 13:18:49.436914 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:18:49.436879    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ns4rd" podUID="ff09df22-abb2-4797-a1c6-6eddaa1a60c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:19:01.026389 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:19:01.026350    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:19:31.028605 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:19:31.028562    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:20:01.026569 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:20:01.026486    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:20:05.089102 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.089068    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"]
Nov 01 13:20:05.092686 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.092657    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.169109 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.169074    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"]
Nov 01 13:20:05.190585 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.190551    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f38349\"/\"openshift-service-ca.crt\""
Nov 01 13:20:05.190585 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.190584    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f38349\"/\"kube-root-ca.crt\""
Nov 01 13:20:05.190819 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.190647    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f38349\"/\"simulator-single-main-f38349-kserve-self-signed-certs\""
Nov 01 13:20:05.190819 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.190720    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f38349\"/\"default-dockercfg-f4mwr\""
Nov 01 13:20:05.230909 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.230868    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-model-cache\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.231074 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.231003    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdds\" (UniqueName: \"kubernetes.io/projected/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-kube-api-access-9kdds\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.231074 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.231046    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tls-certs\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.231195 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.231171    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-home\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.231246 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.231208    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-dshm\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.231328 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.231308    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tmp-dir\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.332466 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.332428    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdds\" (UniqueName: \"kubernetes.io/projected/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-kube-api-access-9kdds\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.332466 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.332463    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tls-certs\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.332759 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.332494    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-home\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.332759 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.332513    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-dshm\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.332759 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.332539    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tmp-dir\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.332759 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.332588    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-model-cache\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.332991 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.332926    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-model-cache\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.332991 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.332939    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-home\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.332991 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.332974    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tmp-dir\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.334954 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.334920    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-dshm\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.335248 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.335229    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tls-certs\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.386743 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.386663    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdds\" (UniqueName: \"kubernetes.io/projected/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-kube-api-access-9kdds\") pod \"simulator-single-main-f38349-kserve-55d7d645d7-x5j2k\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") " pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.405471 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.405437    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:05.780737 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:05.780711    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"]
Nov 01 13:20:05.782914 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:20:05.782882    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecfc0c26_4a23_477e_a6ac_72c4eddcb6d9.slice/crio-3768461f17d388055886f77ed5c22a32cd1bd2b5da9072538b2604ab7d73159b WatchSource:0}: Error finding container 3768461f17d388055886f77ed5c22a32cd1bd2b5da9072538b2604ab7d73159b: Status 404 returned error can't find the container with id 3768461f17d388055886f77ed5c22a32cd1bd2b5da9072538b2604ab7d73159b
Nov 01 13:20:06.319254 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:06.319216    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k" event={"ID":"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9","Type":"ContainerStarted","Data":"0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21"}
Nov 01 13:20:06.319254 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:06.319269    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k" event={"ID":"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9","Type":"ContainerStarted","Data":"3768461f17d388055886f77ed5c22a32cd1bd2b5da9072538b2604ab7d73159b"}
Nov 01 13:20:06.405776 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:06.405738    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:06.427716 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:06.427663    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k" podStartSLOduration=1.427649572 podStartE2EDuration="1.427649572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:20:06.42667141 +0000 UTC m=+3447.933862627" watchObservedRunningTime="2026-11-01 13:20:06.427649572 +0000 UTC m=+3447.934840793"
Nov 01 13:20:15.405969 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:15.405931    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:15.418720 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:15.418696    5725 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:16.398448 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:20:16.398415    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:20:31.028907 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:20:31.028864    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:21:01.026683 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:21:01.026645    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:21:01.065024 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:01.064990    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"]
Nov 01 13:21:16.085445 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.085402    5725 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k" podUID="ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" containerName="main" containerID="cri-o://0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21" gracePeriod=45
Nov 01 13:21:16.332104 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.332079    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:21:16.512465 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.512433    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-model-cache\") pod \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") "
Nov 01 13:21:16.512465 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.512468    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-dshm\") pod \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") "
Nov 01 13:21:16.512731 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.512494    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-kube-api-access-9kdds\" (UniqueName: \"kubernetes.io/projected/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-kube-api-access-9kdds\") pod \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") "
Nov 01 13:21:16.512731 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.512607    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-home\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-home\") pod \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") "
Nov 01 13:21:16.512731 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.512664    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tls-certs\") pod \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") "
Nov 01 13:21:16.512731 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.512728    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tmp-dir\") pod \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\" (UID: \"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9\") "
Nov 01 13:21:16.512923 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.512776    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-model-cache" pod "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" (UID: "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:21:16.512923 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.512862    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-home" pod "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" (UID: "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:21:16.513111 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.513064    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tmp-dir" pod "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" (UID: "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:21:16.513111 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.513103    5725 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-model-cache\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:16.513254 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.513119    5725 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-home\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:16.514825 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.514801    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-dshm" pod "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" (UID: "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:21:16.514957 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.514823    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-kube-api-access-9kdds" pod "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" (UID: "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9"). InnerVolumeSpecName "kube-api-access-9kdds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:21:16.514957 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.514829    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tls-certs" pod "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" (UID: "ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:21:16.614180 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.614143    5725 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tls-certs\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:16.614180 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.614171    5725 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-tmp-dir\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:16.614180 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.614183    5725 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-dshm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:16.614430 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.614195    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kdds\" (UniqueName: \"kubernetes.io/projected/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9-kube-api-access-9kdds\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:16.744449 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.744414    5725 generic.go:356] "Generic (PLEG): container finished" podID="ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" containerID="0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21" exitCode=0
Nov 01 13:21:16.744631 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.744462    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"
Nov 01 13:21:16.744631 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.744519    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k" event={"ID":"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9","Type":"ContainerDied","Data":"0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21"}
Nov 01 13:21:16.744631 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.744569    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k" event={"ID":"ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9","Type":"ContainerDied","Data":"3768461f17d388055886f77ed5c22a32cd1bd2b5da9072538b2604ab7d73159b"}
Nov 01 13:21:16.744631 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.744591    5725 scope.go:122] "RemoveContainer" containerID="0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21"
Nov 01 13:21:16.756265 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.756240    5725 scope.go:122] "RemoveContainer" containerID="0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21"
Nov 01 13:21:16.756601 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:21:16.756579    5725 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21\": container with ID starting with 0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21 not found: ID does not exist" containerID="0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21"
Nov 01 13:21:16.756684 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.756611    5725 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21"} err="failed to get container status \"0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21\": rpc error: code = NotFound desc = could not find container \"0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21\": container with ID starting with 0ac3823dde897c8b9f60249480d36529ba7626422388c555bf1e8228b0431a21 not found: ID does not exist"
Nov 01 13:21:16.899026 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.898925    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"]
Nov 01 13:21:16.945722 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:16.945693    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f38349/simulator-single-main-f38349-kserve-55d7d645d7-x5j2k"]
Nov 01 13:21:18.614801 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:21:18.614768    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" path="/var/lib/kubelet/pods/ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9/volumes"
Nov 01 13:21:31.026712 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:21:31.026665    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:22:01.027007 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:22:01.026965    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:22:31.026234 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:22:31.026197    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:22:39.354182 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:22:39.354149    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:22:39.362972 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:22:39.362946    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:22:39.363507 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:22:39.363486    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:22:39.371974 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:22:39.371950    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:23:01.026465 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:23:01.026429    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:23:31.031230 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:23:31.031188    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:24:01.026460 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:24:01.026423    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:24:31.026425 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:24:31.026341    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:24:38.075685 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.075651    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"]
Nov 01 13:24:38.076160 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.076094    5725 cpu_manager.go:414] "containerMap: removing container" podUID="ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" containerName="main"
Nov 01 13:24:38.076160 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.076108    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" containerName="main"
Nov 01 13:24:38.076255 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.076190    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfc0c26-4a23-477e-a6ac-72c4eddcb6d9" containerName="main"
Nov 01 13:24:38.079384 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.079368    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.176076 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.176021    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jl8dq\""
Nov 01 13:24:38.176076 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.176021    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:24:38.178567 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.178545    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"]
Nov 01 13:24:38.230150 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.223598    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.230150 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.223673    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndw8\" (UniqueName: \"kubernetes.io/projected/f3edc219-c1d9-42db-bae7-5af0e0e342a5-kube-api-access-6ndw8\") pod \"maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.230150 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.223819    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.324472 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.324437    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.324652 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.324499    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.324652 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.324539    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndw8\" (UniqueName: \"kubernetes.io/projected/f3edc219-c1d9-42db-bae7-5af0e0e342a5-kube-api-access-6ndw8\") pod \"maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.325257 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.325221    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.326887 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.326836    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.377803 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.377770    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndw8\" (UniqueName: \"kubernetes.io/projected/f3edc219-c1d9-42db-bae7-5af0e0e342a5-kube-api-access-6ndw8\") pod \"maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.392565 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.392538    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:38.758394 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:38.758371    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"]
Nov 01 13:24:38.760830 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:24:38.760804    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3edc219_c1d9_42db_bae7_5af0e0e342a5.slice/crio-ecea31241512289a8f85153bba1ca0bc336721f4847a28dcca105db91f53861b WatchSource:0}: Error finding container ecea31241512289a8f85153bba1ca0bc336721f4847a28dcca105db91f53861b: Status 404 returned error can't find the container with id ecea31241512289a8f85153bba1ca0bc336721f4847a28dcca105db91f53861b
Nov 01 13:24:39.028555 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:39.028460    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn" event={"ID":"f3edc219-c1d9-42db-bae7-5af0e0e342a5","Type":"ContainerStarted","Data":"ecea31241512289a8f85153bba1ca0bc336721f4847a28dcca105db91f53861b"}
Nov 01 13:24:40.035930 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:40.035892    5725 generic.go:356] "Generic (PLEG): container finished" podID="f3edc219-c1d9-42db-bae7-5af0e0e342a5" containerID="5de6bcad240b7c7e1bc24c100e539dc0a21b7e91993153a3b2b42905f1be5b4a" exitCode=0
Nov 01 13:24:40.036342 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:40.035980    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn" event={"ID":"f3edc219-c1d9-42db-bae7-5af0e0e342a5","Type":"ContainerDied","Data":"5de6bcad240b7c7e1bc24c100e539dc0a21b7e91993153a3b2b42905f1be5b4a"}
Nov 01 13:24:41.180558 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.180528    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:41.353846 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.353764    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-xks-ca\") pod \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") "
Nov 01 13:24:41.353846 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.353799    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3edc219-c1d9-42db-bae7-5af0e0e342a5-kube-api-access-6ndw8\" (UniqueName: \"kubernetes.io/projected/f3edc219-c1d9-42db-bae7-5af0e0e342a5-kube-api-access-6ndw8\") pod \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") "
Nov 01 13:24:41.353846 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.353830    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-openshift-ca\") pod \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\" (UID: \"f3edc219-c1d9-42db-bae7-5af0e0e342a5\") "
Nov 01 13:24:41.354294 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.354268    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-openshift-ca" pod "f3edc219-c1d9-42db-bae7-5af0e0e342a5" (UID: "f3edc219-c1d9-42db-bae7-5af0e0e342a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:24:41.355962 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.355936    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-xks-ca" pod "f3edc219-c1d9-42db-bae7-5af0e0e342a5" (UID: "f3edc219-c1d9-42db-bae7-5af0e0e342a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:24:41.356087 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.356016    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3edc219-c1d9-42db-bae7-5af0e0e342a5-kube-api-access-6ndw8" pod "f3edc219-c1d9-42db-bae7-5af0e0e342a5" (UID: "f3edc219-c1d9-42db-bae7-5af0e0e342a5"). InnerVolumeSpecName "kube-api-access-6ndw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:24:41.454917 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.454883    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:24:41.454917 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.454914    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndw8\" (UniqueName: \"kubernetes.io/projected/f3edc219-c1d9-42db-bae7-5af0e0e342a5-kube-api-access-6ndw8\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:24:41.455144 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:41.454927    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3edc219-c1d9-42db-bae7-5af0e0e342a5-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:24:42.047363 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:42.047328    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn" event={"ID":"f3edc219-c1d9-42db-bae7-5af0e0e342a5","Type":"ContainerDied","Data":"ecea31241512289a8f85153bba1ca0bc336721f4847a28dcca105db91f53861b"}
Nov 01 13:24:42.047548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:42.047383    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecea31241512289a8f85153bba1ca0bc336721f4847a28dcca105db91f53861b"
Nov 01 13:24:42.047548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:42.047344    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"
Nov 01 13:24:57.586372 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:57.586337    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"]
Nov 01 13:24:57.632799 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:57.632763    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f3c81bef-lxbxn"]
Nov 01 13:24:58.615995 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:24:58.615957    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3edc219-c1d9-42db-bae7-5af0e0e342a5" path="/var/lib/kubelet/pods/f3edc219-c1d9-42db-bae7-5af0e0e342a5/volumes"
Nov 01 13:25:01.025760 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:25:01.025714    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:25:31.026154 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:25:31.026114    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:26:01.027538 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:26:01.027460    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:26:31.029629 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:26:31.029596    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:26:52.471787 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.471753    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"]
Nov 01 13:26:52.472315 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.472217    5725 cpu_manager.go:414] "containerMap: removing container" podUID="f3edc219-c1d9-42db-bae7-5af0e0e342a5" containerName="revoke-keys"
Nov 01 13:26:52.472315 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.472229    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3edc219-c1d9-42db-bae7-5af0e0e342a5" containerName="revoke-keys"
Nov 01 13:26:52.472315 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.472302    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3edc219-c1d9-42db-bae7-5af0e0e342a5" containerName="revoke-keys"
Nov 01 13:26:52.475474 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.475457    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.524778 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.524730    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:26:52.528213 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.528177    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"]
Nov 01 13:26:52.571730 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.571696    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jl8dq\""
Nov 01 13:26:52.590180 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.590148    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq8rn\" (UniqueName: \"kubernetes.io/projected/6e7195f9-a345-4d08-9347-9dac37f1641a-kube-api-access-qq8rn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.590300 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.590210    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.590352 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.590300    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.690971 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.690932    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.691178 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.691009    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8rn\" (UniqueName: \"kubernetes.io/projected/6e7195f9-a345-4d08-9347-9dac37f1641a-kube-api-access-qq8rn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.691178 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.691044    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.691737 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.691711    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.693345 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.693317    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.744563 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.744506    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq8rn\" (UniqueName: \"kubernetes.io/projected/6e7195f9-a345-4d08-9347-9dac37f1641a-kube-api-access-qq8rn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:52.788996 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:52.788970    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:53.159669 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:53.159644    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"]
Nov 01 13:26:53.162277 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:26:53.162249    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7195f9_a345_4d08_9347_9dac37f1641a.slice/crio-c6b7ab5fd9482ee9447c0a681b5c025965d9aea4070bcf679c139fd28e1989dd WatchSource:0}: Error finding container c6b7ab5fd9482ee9447c0a681b5c025965d9aea4070bcf679c139fd28e1989dd: Status 404 returned error can't find the container with id c6b7ab5fd9482ee9447c0a681b5c025965d9aea4070bcf679c139fd28e1989dd
Nov 01 13:26:53.796288 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:53.796250    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t" event={"ID":"6e7195f9-a345-4d08-9347-9dac37f1641a","Type":"ContainerStarted","Data":"c2e96c0c99f4621da231acfc58791464f932dd508400e39eddfbf0492659ceff"}
Nov 01 13:26:53.796707 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:53.796307    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t" event={"ID":"6e7195f9-a345-4d08-9347-9dac37f1641a","Type":"ContainerStarted","Data":"c6b7ab5fd9482ee9447c0a681b5c025965d9aea4070bcf679c139fd28e1989dd"}
Nov 01 13:26:53.903299 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:53.903243    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t" podStartSLOduration=1.903227432 podStartE2EDuration="1.903227432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:26:53.903018389 +0000 UTC m=+3855.410209608" watchObservedRunningTime="2026-11-01 13:26:53.903227432 +0000 UTC m=+3855.410418648"
Nov 01 13:26:54.803041 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:54.803005    5725 generic.go:356] "Generic (PLEG): container finished" podID="6e7195f9-a345-4d08-9347-9dac37f1641a" containerID="c2e96c0c99f4621da231acfc58791464f932dd508400e39eddfbf0492659ceff" exitCode=0
Nov 01 13:26:54.803424 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:54.803082    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t" event={"ID":"6e7195f9-a345-4d08-9347-9dac37f1641a","Type":"ContainerDied","Data":"c2e96c0c99f4621da231acfc58791464f932dd508400e39eddfbf0492659ceff"}
Nov 01 13:26:55.951714 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:55.951690    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:56.123190 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.123112    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7195f9-a345-4d08-9347-9dac37f1641a-kube-api-access-qq8rn\" (UniqueName: \"kubernetes.io/projected/6e7195f9-a345-4d08-9347-9dac37f1641a-kube-api-access-qq8rn\") pod \"6e7195f9-a345-4d08-9347-9dac37f1641a\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") "
Nov 01 13:26:56.123351 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.123232    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-xks-ca\") pod \"6e7195f9-a345-4d08-9347-9dac37f1641a\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") "
Nov 01 13:26:56.123351 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.123300    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-openshift-ca\") pod \"6e7195f9-a345-4d08-9347-9dac37f1641a\" (UID: \"6e7195f9-a345-4d08-9347-9dac37f1641a\") "
Nov 01 13:26:56.123706 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.123681    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-openshift-ca" pod "6e7195f9-a345-4d08-9347-9dac37f1641a" (UID: "6e7195f9-a345-4d08-9347-9dac37f1641a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:26:56.125293 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.125268    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7195f9-a345-4d08-9347-9dac37f1641a-kube-api-access-qq8rn" pod "6e7195f9-a345-4d08-9347-9dac37f1641a" (UID: "6e7195f9-a345-4d08-9347-9dac37f1641a"). InnerVolumeSpecName "kube-api-access-qq8rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:26:56.125419 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.125399    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-xks-ca" pod "6e7195f9-a345-4d08-9347-9dac37f1641a" (UID: "6e7195f9-a345-4d08-9347-9dac37f1641a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:26:56.224357 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.224326    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:26:56.224357 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.224352    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq8rn\" (UniqueName: \"kubernetes.io/projected/6e7195f9-a345-4d08-9347-9dac37f1641a-kube-api-access-qq8rn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:26:56.224357 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.224362    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e7195f9-a345-4d08-9347-9dac37f1641a-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:26:56.817334 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.817302    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"
Nov 01 13:26:56.817529 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.817302    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t" event={"ID":"6e7195f9-a345-4d08-9347-9dac37f1641a","Type":"ContainerDied","Data":"c6b7ab5fd9482ee9447c0a681b5c025965d9aea4070bcf679c139fd28e1989dd"}
Nov 01 13:26:56.817529 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:26:56.817422    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b7ab5fd9482ee9447c0a681b5c025965d9aea4070bcf679c139fd28e1989dd"
Nov 01 13:27:01.028701 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:27:01.028661    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:27:13.977033 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:13.976993    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"]
Nov 01 13:27:14.024072 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:14.024015    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-624dd4-5lx7t"]
Nov 01 13:27:14.615695 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:14.615660    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7195f9-a345-4d08-9347-9dac37f1641a" path="/var/lib/kubelet/pods/6e7195f9-a345-4d08-9347-9dac37f1641a/volumes"
Nov 01 13:27:31.027275 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:27:31.027195    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:27:39.397937 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:39.397899    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:27:39.405623 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:39.405604    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:27:39.406764 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:39.406743    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dnj2m_90780cda-beee-4fe4-b767-0f28893febe4/extract/0.log"
Nov 01 13:27:39.414787 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:39.414767    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-slc7h_9b41f9fc-afce-464d-a18c-330694f80140/console-operator/2.log"
Nov 01 13:27:50.825942 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.825901    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"]
Nov 01 13:27:50.826362 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.826353    5725 cpu_manager.go:414] "containerMap: removing container" podUID="6e7195f9-a345-4d08-9347-9dac37f1641a" containerName="revoke-keys"
Nov 01 13:27:50.826406 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.826365    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7195f9-a345-4d08-9347-9dac37f1641a" containerName="revoke-keys"
Nov 01 13:27:50.826443 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.826432    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7195f9-a345-4d08-9347-9dac37f1641a" containerName="revoke-keys"
Nov 01 13:27:50.829591 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.829573    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:50.870116 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.870038    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pd6dq\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:50.870311 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.870153    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pd6dq\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:50.870311 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.870212    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgk6\" (UniqueName: \"kubernetes.io/projected/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-kube-api-access-9zgk6\") pod \"maas-api-revoke-sub-f03629db-pd6dq\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:50.879446 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.879335    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:27:50.882999 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.882969    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"]
Nov 01 13:27:50.926129 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.926102    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jl8dq\""
Nov 01 13:27:50.971381 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.971348    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pd6dq\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:50.971578 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.971388    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pd6dq\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:50.971578 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.971424    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgk6\" (UniqueName: \"kubernetes.io/projected/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-kube-api-access-9zgk6\") pod \"maas-api-revoke-sub-f03629db-pd6dq\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:50.972140 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.972112    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pd6dq\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:50.973772 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:50.973749    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pd6dq\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:51.026663 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:51.026635    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgk6\" (UniqueName: \"kubernetes.io/projected/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-kube-api-access-9zgk6\") pod \"maas-api-revoke-sub-f03629db-pd6dq\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:51.142957 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:51.142875    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:51.514661 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:51.514516    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"]
Nov 01 13:27:51.517244 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:27:51.517216    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7ee80d_6905_4cdb_8fbf_8f8787a15ee5.slice/crio-d9f8077429e80852459b3ae6346de0356915a3adaf10c6472b3e4ef6ff346665 WatchSource:0}: Error finding container d9f8077429e80852459b3ae6346de0356915a3adaf10c6472b3e4ef6ff346665: Status 404 returned error can't find the container with id d9f8077429e80852459b3ae6346de0356915a3adaf10c6472b3e4ef6ff346665
Nov 01 13:27:52.164140 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:52.164035    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq" event={"ID":"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5","Type":"ContainerStarted","Data":"34f5ae159eb132578dba964248dec6f4e46be18be70463620d007510cf7fe8a7"}
Nov 01 13:27:52.164140 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:52.164102    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq" event={"ID":"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5","Type":"ContainerStarted","Data":"d9f8077429e80852459b3ae6346de0356915a3adaf10c6472b3e4ef6ff346665"}
Nov 01 13:27:52.274720 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:52.274660    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq" podStartSLOduration=2.274647523 podStartE2EDuration="2.274647523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:27:52.272591545 +0000 UTC m=+3913.779782766" watchObservedRunningTime="2026-11-01 13:27:52.274647523 +0000 UTC m=+3913.781838734"
Nov 01 13:27:53.171669 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:53.171629    5725 generic.go:356] "Generic (PLEG): container finished" podID="eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5" containerID="34f5ae159eb132578dba964248dec6f4e46be18be70463620d007510cf7fe8a7" exitCode=0
Nov 01 13:27:53.172177 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:53.171709    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq" event={"ID":"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5","Type":"ContainerDied","Data":"34f5ae159eb132578dba964248dec6f4e46be18be70463620d007510cf7fe8a7"}
Nov 01 13:27:54.316252 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.316225    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:54.404626 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.404589    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-xks-ca\") pod \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") "
Nov 01 13:27:54.404837 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.404790    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-kube-api-access-9zgk6\" (UniqueName: \"kubernetes.io/projected/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-kube-api-access-9zgk6\") pod \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") "
Nov 01 13:27:54.404837 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.404837    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-openshift-ca\") pod \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\" (UID: \"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5\") "
Nov 01 13:27:54.405257 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.405234    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-openshift-ca" pod "eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5" (UID: "eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:27:54.406785 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.406764    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-kube-api-access-9zgk6" pod "eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5" (UID: "eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5"). InnerVolumeSpecName "kube-api-access-9zgk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:27:54.406879 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.406801    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-xks-ca" pod "eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5" (UID: "eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:27:54.505505 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.505466    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zgk6\" (UniqueName: \"kubernetes.io/projected/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-kube-api-access-9zgk6\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:54.505505 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.505499    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:54.505732 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:54.505512    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:55.184027 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:55.183990    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq" event={"ID":"eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5","Type":"ContainerDied","Data":"d9f8077429e80852459b3ae6346de0356915a3adaf10c6472b3e4ef6ff346665"}
Nov 01 13:27:55.184219 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:55.184029    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pd6dq"
Nov 01 13:27:55.184219 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:27:55.184045    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9f8077429e80852459b3ae6346de0356915a3adaf10c6472b3e4ef6ff346665"
Nov 01 13:28:01.026877 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:28:01.026834    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:28:09.716549 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.716514    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"]
Nov 01 13:28:09.717176 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.717154    5725 cpu_manager.go:414] "containerMap: removing container" podUID="eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5" containerName="revoke-keys"
Nov 01 13:28:09.717274 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.717179    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5" containerName="revoke-keys"
Nov 01 13:28:09.717342 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.717323    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5" containerName="revoke-keys"
Nov 01 13:28:09.722243 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.722212    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:09.806429 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.806396    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"]
Nov 01 13:28:09.854532 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.854355    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:28:09.854532 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.854499    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jl8dq\""
Nov 01 13:28:09.855486 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.855462    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-s797h\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:09.902092 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.855689    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-s797h\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:09.903306 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:09.903256    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfdm\" (UniqueName: \"kubernetes.io/projected/de6edacd-1cc2-4226-91ce-b811b62b69b7-kube-api-access-jmfdm\") pod \"maas-api-revoke-sub-0165d7d2-s797h\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:10.010373 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:10.008596    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfdm\" (UniqueName: \"kubernetes.io/projected/de6edacd-1cc2-4226-91ce-b811b62b69b7-kube-api-access-jmfdm\") pod \"maas-api-revoke-sub-0165d7d2-s797h\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:10.010373 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:10.008692    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-s797h\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:10.010373 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:10.008734    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-s797h\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:10.010373 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:10.009637    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-s797h\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:10.020192 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:10.019440    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-s797h\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:10.113491 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:10.113453    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfdm\" (UniqueName: \"kubernetes.io/projected/de6edacd-1cc2-4226-91ce-b811b62b69b7-kube-api-access-jmfdm\") pod \"maas-api-revoke-sub-0165d7d2-s797h\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:10.350296 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:10.350200    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:10.826346 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:10.826319    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"]
Nov 01 13:28:11.331919 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:11.331885    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h" event={"ID":"de6edacd-1cc2-4226-91ce-b811b62b69b7","Type":"ContainerStarted","Data":"26b439181940230dfb87167cb766f168a6eb6c403f122e3e27af57f02bd9d5fc"}
Nov 01 13:28:12.339014 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:12.338981    5725 generic.go:356] "Generic (PLEG): container finished" podID="de6edacd-1cc2-4226-91ce-b811b62b69b7" containerID="d8a28653ae929366438c8425f2c78c1c14c1c7e0e8af87876e69de96761ee1f2" exitCode=0
Nov 01 13:28:12.339424 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:12.339072    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h" event={"ID":"de6edacd-1cc2-4226-91ce-b811b62b69b7","Type":"ContainerDied","Data":"d8a28653ae929366438c8425f2c78c1c14c1c7e0e8af87876e69de96761ee1f2"}
Nov 01 13:28:13.481419 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.481389    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:13.551107 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.551072    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6edacd-1cc2-4226-91ce-b811b62b69b7-kube-api-access-jmfdm\" (UniqueName: \"kubernetes.io/projected/de6edacd-1cc2-4226-91ce-b811b62b69b7-kube-api-access-jmfdm\") pod \"de6edacd-1cc2-4226-91ce-b811b62b69b7\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") "
Nov 01 13:28:13.551276 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.551129    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-xks-ca\") pod \"de6edacd-1cc2-4226-91ce-b811b62b69b7\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") "
Nov 01 13:28:13.553212 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.553182    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-xks-ca" pod "de6edacd-1cc2-4226-91ce-b811b62b69b7" (UID: "de6edacd-1cc2-4226-91ce-b811b62b69b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:28:13.553369 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.553302    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6edacd-1cc2-4226-91ce-b811b62b69b7-kube-api-access-jmfdm" pod "de6edacd-1cc2-4226-91ce-b811b62b69b7" (UID: "de6edacd-1cc2-4226-91ce-b811b62b69b7"). InnerVolumeSpecName "kube-api-access-jmfdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:28:13.651823 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.651746    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-openshift-ca\") pod \"de6edacd-1cc2-4226-91ce-b811b62b69b7\" (UID: \"de6edacd-1cc2-4226-91ce-b811b62b69b7\") "
Nov 01 13:28:13.651983 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.651924    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmfdm\" (UniqueName: \"kubernetes.io/projected/de6edacd-1cc2-4226-91ce-b811b62b69b7-kube-api-access-jmfdm\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:28:13.651983 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.651937    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:28:13.652116 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.652092    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-openshift-ca" pod "de6edacd-1cc2-4226-91ce-b811b62b69b7" (UID: "de6edacd-1cc2-4226-91ce-b811b62b69b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:28:13.752840 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:13.752804    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de6edacd-1cc2-4226-91ce-b811b62b69b7-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:28:14.353000 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:14.352968    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h"
Nov 01 13:28:14.353186 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:14.352972    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s797h" event={"ID":"de6edacd-1cc2-4226-91ce-b811b62b69b7","Type":"ContainerDied","Data":"26b439181940230dfb87167cb766f168a6eb6c403f122e3e27af57f02bd9d5fc"}
Nov 01 13:28:14.353186 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:14.353096    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26b439181940230dfb87167cb766f168a6eb6c403f122e3e27af57f02bd9d5fc"
Nov 01 13:28:21.338474 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.338442    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"]
Nov 01 13:28:21.338865 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.338836    5725 cpu_manager.go:414] "containerMap: removing container" podUID="de6edacd-1cc2-4226-91ce-b811b62b69b7" containerName="revoke-keys"
Nov 01 13:28:21.338865 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.338847    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6edacd-1cc2-4226-91ce-b811b62b69b7" containerName="revoke-keys"
Nov 01 13:28:21.338967 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.338955    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6edacd-1cc2-4226-91ce-b811b62b69b7" containerName="revoke-keys"
Nov 01 13:28:21.342223 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.342206    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.395485 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.395453    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"]
Nov 01 13:28:21.410260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.410228    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhkb\" (UniqueName: \"kubernetes.io/projected/9b65525f-1749-4a06-8346-5d1ebd3ac56e-kube-api-access-wlhkb\") pod \"maas-api-revoke-sub-6b19338f-phdhd\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.410397 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.410336    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-phdhd\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.410397 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.410394    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-phdhd\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.438721 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.438675    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:28:21.438870 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.438724    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jl8dq\""
Nov 01 13:28:21.511195 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.511165    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhkb\" (UniqueName: \"kubernetes.io/projected/9b65525f-1749-4a06-8346-5d1ebd3ac56e-kube-api-access-wlhkb\") pod \"maas-api-revoke-sub-6b19338f-phdhd\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.511381 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.511223    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-phdhd\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.511381 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.511283    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-phdhd\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.511965 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.511937    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-phdhd\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.513823 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.513798    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-phdhd\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.564902 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.564868    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhkb\" (UniqueName: \"kubernetes.io/projected/9b65525f-1749-4a06-8346-5d1ebd3ac56e-kube-api-access-wlhkb\") pod \"maas-api-revoke-sub-6b19338f-phdhd\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:21.655235 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:21.655159    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:22.021244 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:22.021211    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"]
Nov 01 13:28:22.025455 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:28:22.025423    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b65525f_1749_4a06_8346_5d1ebd3ac56e.slice/crio-de82e9618afde5d5b2e40e272351da098325ab93bdc7e1c275a502cd31b1c704 WatchSource:0}: Error finding container de82e9618afde5d5b2e40e272351da098325ab93bdc7e1c275a502cd31b1c704: Status 404 returned error can't find the container with id de82e9618afde5d5b2e40e272351da098325ab93bdc7e1c275a502cd31b1c704
Nov 01 13:28:22.404189 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:22.404153    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd" event={"ID":"9b65525f-1749-4a06-8346-5d1ebd3ac56e","Type":"ContainerStarted","Data":"8a5bdd35628465cc57685b553b94604f1251bafd8e376576731efc0990d4eeb9"}
Nov 01 13:28:22.404583 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:22.404204    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd" event={"ID":"9b65525f-1749-4a06-8346-5d1ebd3ac56e","Type":"ContainerStarted","Data":"de82e9618afde5d5b2e40e272351da098325ab93bdc7e1c275a502cd31b1c704"}
Nov 01 13:28:22.513439 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:22.513377    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd" podStartSLOduration=1.51336394 podStartE2EDuration="1.51336394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:28:22.510705123 +0000 UTC m=+3944.017896340" watchObservedRunningTime="2026-11-01 13:28:22.51336394 +0000 UTC m=+3944.020555154"
Nov 01 13:28:23.410718 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:23.410685    5725 generic.go:356] "Generic (PLEG): container finished" podID="9b65525f-1749-4a06-8346-5d1ebd3ac56e" containerID="8a5bdd35628465cc57685b553b94604f1251bafd8e376576731efc0990d4eeb9" exitCode=0
Nov 01 13:28:23.411145 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:23.410740    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd" event={"ID":"9b65525f-1749-4a06-8346-5d1ebd3ac56e","Type":"ContainerDied","Data":"8a5bdd35628465cc57685b553b94604f1251bafd8e376576731efc0990d4eeb9"}
Nov 01 13:28:24.552406 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.552380    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:24.639748 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.639718    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-openshift-ca\") pod \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") "
Nov 01 13:28:24.639971 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.639770    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b65525f-1749-4a06-8346-5d1ebd3ac56e-kube-api-access-wlhkb\" (UniqueName: \"kubernetes.io/projected/9b65525f-1749-4a06-8346-5d1ebd3ac56e-kube-api-access-wlhkb\") pod \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") "
Nov 01 13:28:24.639971 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.639852    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-xks-ca\") pod \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\" (UID: \"9b65525f-1749-4a06-8346-5d1ebd3ac56e\") "
Nov 01 13:28:24.640151 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.640121    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-openshift-ca" pod "9b65525f-1749-4a06-8346-5d1ebd3ac56e" (UID: "9b65525f-1749-4a06-8346-5d1ebd3ac56e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:28:24.640247 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.640224    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-openshift-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:28:24.642432 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.642404    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-xks-ca" pod "9b65525f-1749-4a06-8346-5d1ebd3ac56e" (UID: "9b65525f-1749-4a06-8346-5d1ebd3ac56e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:28:24.642553 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.642446    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b65525f-1749-4a06-8346-5d1ebd3ac56e-kube-api-access-wlhkb" pod "9b65525f-1749-4a06-8346-5d1ebd3ac56e" (UID: "9b65525f-1749-4a06-8346-5d1ebd3ac56e"). InnerVolumeSpecName "kube-api-access-wlhkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:28:24.741613 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.741582    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlhkb\" (UniqueName: \"kubernetes.io/projected/9b65525f-1749-4a06-8346-5d1ebd3ac56e-kube-api-access-wlhkb\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:28:24.741613 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:24.741611    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b65525f-1749-4a06-8346-5d1ebd3ac56e-maas-api-xks-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:28:25.423395 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:25.423360    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd"
Nov 01 13:28:25.423395 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:25.423382    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-phdhd" event={"ID":"9b65525f-1749-4a06-8346-5d1ebd3ac56e","Type":"ContainerDied","Data":"de82e9618afde5d5b2e40e272351da098325ab93bdc7e1c275a502cd31b1c704"}
Nov 01 13:28:25.423621 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:25.423426    5725 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de82e9618afde5d5b2e40e272351da098325ab93bdc7e1c275a502cd31b1c704"
Nov 01 13:28:31.026227 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:28:31.026191    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:28:33.440108 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:33.440071    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f9995dd6f-pppw8_a563ddad-4fab-4b9f-9e01-c325efed519a/authorino/0.log"
Nov 01 13:28:36.144227 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:36.144183    5725 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-hs25d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 13:28:36.144883 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:36.144847    5725 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-hs25d" podUID="fbe0ab95-980d-496a-8437-b3c6e552f9bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:28:44.089708 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:44.089673    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tlnw9_ac494857-512e-4d8d-ae3e-127a0ededaf2/manager/0.log"
Nov 01 13:28:44.356975 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:44.356899    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dbmkl_5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa/manager/0.log"
Nov 01 13:28:45.266413 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:45.266383    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-rrdwb_7234148f-b03b-4e54-8e88-61d41c027559/manager/0.log"
Nov 01 13:28:45.521728 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:45.521659    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vfwh2_6a039198-640d-40f0-9507-2968a5413a77/proxy/0.log"
Nov 01 13:28:46.699597 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:46.699557    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-s797h_de6edacd-1cc2-4226-91ce-b811b62b69b7/revoke-keys/0.log"
Nov 01 13:28:47.110260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:47.110227    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-phdhd_9b65525f-1749-4a06-8346-5d1ebd3ac56e/revoke-keys/0.log"
Nov 01 13:28:47.324493 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:47.324457    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-pd6dq_eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5/revoke-keys/0.log"
Nov 01 13:28:47.537384 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:47.537354    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-ctvqj_485e0b05-d222-4063-82fe-5211239527db/postgres/0.log"
Nov 01 13:28:48.961881 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:48.961792    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f9995dd6f-pppw8_a563ddad-4fab-4b9f-9e01-c325efed519a/authorino/0.log"
Nov 01 13:28:49.360217 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:49.360189    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5d2lp_0d67bd5f-fba2-46dd-a457-124bf4b71a5f/manager/0.log"
Nov 01 13:28:49.575027 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:49.574997    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zvk2f_6483676f-1bb7-4013-8246-aa877be51bb5/manager/0.log"
Nov 01 13:28:50.028025 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:50.027995    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bx6xs_838325b1-57bd-4388-a34f-e524c094fcc1/manager/0.log"
Nov 01 13:28:50.533441 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:50.533400    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4nhzf_73be14a6-2a72-424c-9fc9-4f5cc6fd9e53/manager/0.log"
Nov 01 13:28:51.304227 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:51.304188    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9clm8lm_7774276a-5194-4bbc-bb2b-7eb93e7fa6c2/istio-proxy/0.log"
Nov 01 13:28:53.669670 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:53.669638    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-hs25d_fbe0ab95-980d-496a-8437-b3c6e552f9bc/istio-proxy/0.log"
Nov 01 13:28:53.891327 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:53.891298    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-ns4rd_ff09df22-abb2-4797-a1c6-6eddaa1a60c0/istio-proxy/0.log"
Nov 01 13:28:54.091865 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:54.091831    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-rxn6r_63317aed-7a2c-41fe-8eee-ece29d7326cc/istio-proxy/0.log"
Nov 01 13:28:54.309535 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:54.309504    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl_de6b7627-2b0c-416b-b3fe-6f6d03e09089/istio-proxy/0.log"
Nov 01 13:28:54.920950 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:54.920919    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79dd7d46d9-xfhjz_426e64a6-ecf0-471e-b130-15f85716445f/router/0.log"
Nov 01 13:28:58.019561 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:28:58.019530    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f9995dd6f-pppw8_a563ddad-4fab-4b9f-9e01-c325efed519a/authorino/0.log"
Nov 01 13:29:01.026448 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:29:01.026410    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:29:07.271976 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:07.271938    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tlnw9_ac494857-512e-4d8d-ae3e-127a0ededaf2/manager/0.log"
Nov 01 13:29:07.542651 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:07.542557    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dbmkl_5cdd86a2-a4f9-4786-99e9-8d7d5e6bfdaa/manager/0.log"
Nov 01 13:29:08.238065 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:08.238024    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-rrdwb_7234148f-b03b-4e54-8e88-61d41c027559/manager/0.log"
Nov 01 13:29:08.489817 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:08.489728    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vfwh2_6a039198-640d-40f0-9507-2968a5413a77/proxy/0.log"
Nov 01 13:29:09.474505 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:09.474480    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-s797h_de6edacd-1cc2-4226-91ce-b811b62b69b7/revoke-keys/0.log"
Nov 01 13:29:09.681439 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:09.681402    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-phdhd_9b65525f-1749-4a06-8346-5d1ebd3ac56e/revoke-keys/0.log"
Nov 01 13:29:09.898622 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:09.898531    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-pd6dq_eb7ee80d-6905-4cdb-8fbf-8f8787a15ee5/revoke-keys/0.log"
Nov 01 13:29:10.107830 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:10.107801    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-ctvqj_485e0b05-d222-4063-82fe-5211239527db/postgres/0.log"
Nov 01 13:29:11.168154 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:11.168120    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f9995dd6f-pppw8_a563ddad-4fab-4b9f-9e01-c325efed519a/authorino/0.log"
Nov 01 13:29:11.544826 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:11.544798    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5d2lp_0d67bd5f-fba2-46dd-a457-124bf4b71a5f/manager/0.log"
Nov 01 13:29:11.768989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:11.768959    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zvk2f_6483676f-1bb7-4013-8246-aa877be51bb5/manager/0.log"
Nov 01 13:29:12.189871 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:12.189843    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bx6xs_838325b1-57bd-4388-a34f-e524c094fcc1/manager/0.log"
Nov 01 13:29:12.663280 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:12.663246    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4nhzf_73be14a6-2a72-424c-9fc9-4f5cc6fd9e53/manager/0.log"
Nov 01 13:29:13.255974 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:13.255938    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9clm8lm_7774276a-5194-4bbc-bb2b-7eb93e7fa6c2/istio-proxy/0.log"
Nov 01 13:29:15.599287 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:15.599258    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-hs25d_fbe0ab95-980d-496a-8437-b3c6e552f9bc/istio-proxy/0.log"
Nov 01 13:29:15.821441 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:15.821411    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-ns4rd_ff09df22-abb2-4797-a1c6-6eddaa1a60c0/istio-proxy/0.log"
Nov 01 13:29:16.043445 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:16.043418    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-rxn6r_63317aed-7a2c-41fe-8eee-ece29d7326cc/istio-proxy/0.log"
Nov 01 13:29:16.258571 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:16.258535    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-668d64f647-j6ljl_de6b7627-2b0c-416b-b3fe-6f6d03e09089/istio-proxy/0.log"
Nov 01 13:29:16.867738 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:29:16.867706    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79dd7d46d9-xfhjz_426e64a6-ecf0-471e-b130-15f85716445f/router/0.log"
Nov 01 13:29:31.026916 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:29:31.026813    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:30:00.165057 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.165022    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"]
Nov 01 13:30:00.165615 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.165486    5725 cpu_manager.go:414] "containerMap: removing container" podUID="9b65525f-1749-4a06-8346-5d1ebd3ac56e" containerName="revoke-keys"
Nov 01 13:30:00.165615 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.165497    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b65525f-1749-4a06-8346-5d1ebd3ac56e" containerName="revoke-keys"
Nov 01 13:30:00.165615 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.165574    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b65525f-1749-4a06-8346-5d1ebd3ac56e" containerName="revoke-keys"
Nov 01 13:30:00.168748 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.168726    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:00.218157 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.218126    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jl8dq\""
Nov 01 13:30:00.244280 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.244235    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw2hn\" (UniqueName: \"kubernetes.io/projected/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-kube-api-access-jw2hn\") pod \"maas-api-key-cleanup-29892330-542wx\" (UID: \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:00.244479 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.244339    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892330-542wx\" (UID: \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:00.267334 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.267302    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"]
Nov 01 13:30:00.345274 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.345241    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892330-542wx\" (UID: \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:00.345462 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.345325    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw2hn\" (UniqueName: \"kubernetes.io/projected/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-kube-api-access-jw2hn\") pod \"maas-api-key-cleanup-29892330-542wx\" (UID: \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:00.345891 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.345871    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892330-542wx\" (UID: \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:00.399616 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.399578    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw2hn\" (UniqueName: \"kubernetes.io/projected/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-kube-api-access-jw2hn\") pod \"maas-api-key-cleanup-29892330-542wx\" (UID: \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:00.482443 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.482419    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:00.854283 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.854203    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"]
Nov 01 13:30:00.862041 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:30:00.862012    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4048c126_4b81_4dcf_8a25_6d39e3dba9fb.slice/crio-15c93d45f243152ab559e98a901bb298bfa7dfb27e7e40c1a245055309e44c45 WatchSource:0}: Error finding container 15c93d45f243152ab559e98a901bb298bfa7dfb27e7e40c1a245055309e44c45: Status 404 returned error can't find the container with id 15c93d45f243152ab559e98a901bb298bfa7dfb27e7e40c1a245055309e44c45
Nov 01 13:30:00.980146 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:00.980109    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" event={"ID":"4048c126-4b81-4dcf-8a25-6d39e3dba9fb","Type":"ContainerStarted","Data":"15c93d45f243152ab559e98a901bb298bfa7dfb27e7e40c1a245055309e44c45"}
Nov 01 13:30:01.029253 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:30:01.029217    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:30:01.987710 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:01.987672    5725 generic.go:356] "Generic (PLEG): container finished" podID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerID="1f77ab72ce2fb412bcde44ba95ff82a2ac6700504c4507636c95830070233e13" exitCode=60
Nov 01 13:30:01.988140 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:01.987755    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" event={"ID":"4048c126-4b81-4dcf-8a25-6d39e3dba9fb","Type":"ContainerDied","Data":"1f77ab72ce2fb412bcde44ba95ff82a2ac6700504c4507636c95830070233e13"}
Nov 01 13:30:01.988140 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:01.988039    5725 scope.go:122] "RemoveContainer" containerID="1f77ab72ce2fb412bcde44ba95ff82a2ac6700504c4507636c95830070233e13"
Nov 01 13:30:02.996096 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:02.995987    5725 generic.go:356] "Generic (PLEG): container finished" podID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerID="90451704b5a21740296b7fcfc7896de796eda62a72676e8229979fb954f0754c" exitCode=60
Nov 01 13:30:02.996096 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:02.996069    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" event={"ID":"4048c126-4b81-4dcf-8a25-6d39e3dba9fb","Type":"ContainerDied","Data":"90451704b5a21740296b7fcfc7896de796eda62a72676e8229979fb954f0754c"}
Nov 01 13:30:02.996615 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:02.996116    5725 scope.go:122] "RemoveContainer" containerID="1f77ab72ce2fb412bcde44ba95ff82a2ac6700504c4507636c95830070233e13"
Nov 01 13:30:02.996615 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:02.996351    5725 scope.go:122] "RemoveContainer" containerID="90451704b5a21740296b7fcfc7896de796eda62a72676e8229979fb954f0754c"
Nov 01 13:30:02.996615 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:30:02.996599    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892330-542wx_odh-ai-gateway-infra(4048c126-4b81-4dcf-8a25-6d39e3dba9fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb"
Nov 01 13:30:04.004431 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:04.004404    5725 scope.go:122] "RemoveContainer" containerID="90451704b5a21740296b7fcfc7896de796eda62a72676e8229979fb954f0754c"
Nov 01 13:30:04.004840 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:30:04.004630    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892330-542wx_odh-ai-gateway-infra(4048c126-4b81-4dcf-8a25-6d39e3dba9fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb"
Nov 01 13:30:14.609930 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:14.609895    5725 scope.go:122] "RemoveContainer" containerID="90451704b5a21740296b7fcfc7896de796eda62a72676e8229979fb954f0754c"
Nov 01 13:30:15.079433 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:15.079394    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" event={"ID":"4048c126-4b81-4dcf-8a25-6d39e3dba9fb","Type":"ContainerStarted","Data":"395f03433a4c4c03e43e7b6a61b8f3af92f10c526437cf95f4b90f7fcd24280e"}
Nov 01 13:30:15.187791 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:15.187687    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" podStartSLOduration=15.18767473 podStartE2EDuration="15.18767473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:30:15.186463142 +0000 UTC m=+4056.693654359" watchObservedRunningTime="2026-11-01 13:30:15.18767473 +0000 UTC m=+4056.694865947"
Nov 01 13:30:16.087460 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:16.087422    5725 generic.go:356] "Generic (PLEG): container finished" podID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerID="395f03433a4c4c03e43e7b6a61b8f3af92f10c526437cf95f4b90f7fcd24280e" exitCode=60
Nov 01 13:30:16.087460 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:16.087464    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" event={"ID":"4048c126-4b81-4dcf-8a25-6d39e3dba9fb","Type":"ContainerDied","Data":"395f03433a4c4c03e43e7b6a61b8f3af92f10c526437cf95f4b90f7fcd24280e"}
Nov 01 13:30:16.087981 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:16.087541    5725 scope.go:122] "RemoveContainer" containerID="90451704b5a21740296b7fcfc7896de796eda62a72676e8229979fb954f0754c"
Nov 01 13:30:16.087981 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:16.087893    5725 scope.go:122] "RemoveContainer" containerID="395f03433a4c4c03e43e7b6a61b8f3af92f10c526437cf95f4b90f7fcd24280e"
Nov 01 13:30:16.088185 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:30:16.088165    5725 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892330-542wx_odh-ai-gateway-infra(4048c126-4b81-4dcf-8a25-6d39e3dba9fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb"
Nov 01 13:30:16.203599 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:16.203564    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"]
Nov 01 13:30:17.233042 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:17.233014    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:17.305889 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:17.305856    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-maas-api-service-ca\") pod \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\" (UID: \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\") "
Nov 01 13:30:17.306126 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:17.305953    5725 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-kube-api-access-jw2hn\" (UniqueName: \"kubernetes.io/projected/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-kube-api-access-jw2hn\") pod \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\" (UID: \"4048c126-4b81-4dcf-8a25-6d39e3dba9fb\") "
Nov 01 13:30:17.306330 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:17.306305    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-maas-api-service-ca" pod "4048c126-4b81-4dcf-8a25-6d39e3dba9fb" (UID: "4048c126-4b81-4dcf-8a25-6d39e3dba9fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:30:17.308027 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:17.308007    5725 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-kube-api-access-jw2hn" pod "4048c126-4b81-4dcf-8a25-6d39e3dba9fb" (UID: "4048c126-4b81-4dcf-8a25-6d39e3dba9fb"). InnerVolumeSpecName "kube-api-access-jw2hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:30:17.406939 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:17.406847    5725 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw2hn\" (UniqueName: \"kubernetes.io/projected/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-kube-api-access-jw2hn\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:30:17.406939 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:17.406878    5725 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4048c126-4b81-4dcf-8a25-6d39e3dba9fb-maas-api-service-ca\") on node \"ip-10-0-138-76.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:30:18.102515 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:18.102483    5725 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"
Nov 01 13:30:18.102515 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:18.102492    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx" event={"ID":"4048c126-4b81-4dcf-8a25-6d39e3dba9fb","Type":"ContainerDied","Data":"15c93d45f243152ab559e98a901bb298bfa7dfb27e7e40c1a245055309e44c45"}
Nov 01 13:30:18.102763 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:18.102559    5725 scope.go:122] "RemoveContainer" containerID="395f03433a4c4c03e43e7b6a61b8f3af92f10c526437cf95f4b90f7fcd24280e"
Nov 01 13:30:18.256989 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:18.256953    5725 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"]
Nov 01 13:30:18.303844 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:18.303805    5725 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892330-542wx"]
Nov 01 13:30:18.617549 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:18.617515    5725 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" path="/var/lib/kubelet/pods/4048c126-4b81-4dcf-8a25-6d39e3dba9fb/volumes"
Nov 01 13:30:22.060998 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.060966    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9plmq/must-gather-q2rl2"]
Nov 01 13:30:22.061414 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.061380    5725 cpu_manager.go:414] "containerMap: removing container" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerName="cleanup"
Nov 01 13:30:22.061414 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.061392    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerName="cleanup"
Nov 01 13:30:22.061414 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.061415    5725 cpu_manager.go:414] "containerMap: removing container" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerName="cleanup"
Nov 01 13:30:22.061548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.061421    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerName="cleanup"
Nov 01 13:30:22.061548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.061489    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerName="cleanup"
Nov 01 13:30:22.061548 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.061501    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerName="cleanup"
Nov 01 13:30:22.061646 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.061574    5725 cpu_manager.go:414] "containerMap: removing container" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerName="cleanup"
Nov 01 13:30:22.061646 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.061579    5725 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerName="cleanup"
Nov 01 13:30:22.061646 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.061627    5725 memory_manager.go:365] "RemoveStaleState removing state" podUID="4048c126-4b81-4dcf-8a25-6d39e3dba9fb" containerName="cleanup"
Nov 01 13:30:22.066122 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.066103    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9plmq/must-gather-q2rl2"
Nov 01 13:30:22.116979 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.116946    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9plmq/must-gather-q2rl2"]
Nov 01 13:30:22.154565 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.154530    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/782401f2-3cb5-4799-be6c-92e75186b676-must-gather-output\") pod \"must-gather-q2rl2\" (UID: \"782401f2-3cb5-4799-be6c-92e75186b676\") " pod="openshift-must-gather-9plmq/must-gather-q2rl2"
Nov 01 13:30:22.154729 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.154580    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr56b\" (UniqueName: \"kubernetes.io/projected/782401f2-3cb5-4799-be6c-92e75186b676-kube-api-access-gr56b\") pod \"must-gather-q2rl2\" (UID: \"782401f2-3cb5-4799-be6c-92e75186b676\") " pod="openshift-must-gather-9plmq/must-gather-q2rl2"
Nov 01 13:30:22.161545 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.161524    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9plmq\"/\"openshift-service-ca.crt\""
Nov 01 13:30:22.163119 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.163097    5725 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9plmq\"/\"default-dockercfg-k7gkk\""
Nov 01 13:30:22.163177 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.163102    5725 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9plmq\"/\"kube-root-ca.crt\""
Nov 01 13:30:22.256132 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.256097    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr56b\" (UniqueName: \"kubernetes.io/projected/782401f2-3cb5-4799-be6c-92e75186b676-kube-api-access-gr56b\") pod \"must-gather-q2rl2\" (UID: \"782401f2-3cb5-4799-be6c-92e75186b676\") " pod="openshift-must-gather-9plmq/must-gather-q2rl2"
Nov 01 13:30:22.256309 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.256199    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/782401f2-3cb5-4799-be6c-92e75186b676-must-gather-output\") pod \"must-gather-q2rl2\" (UID: \"782401f2-3cb5-4799-be6c-92e75186b676\") " pod="openshift-must-gather-9plmq/must-gather-q2rl2"
Nov 01 13:30:22.256535 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.256516    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/782401f2-3cb5-4799-be6c-92e75186b676-must-gather-output\") pod \"must-gather-q2rl2\" (UID: \"782401f2-3cb5-4799-be6c-92e75186b676\") " pod="openshift-must-gather-9plmq/must-gather-q2rl2"
Nov 01 13:30:22.309721 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.309689    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr56b\" (UniqueName: \"kubernetes.io/projected/782401f2-3cb5-4799-be6c-92e75186b676-kube-api-access-gr56b\") pod \"must-gather-q2rl2\" (UID: \"782401f2-3cb5-4799-be6c-92e75186b676\") " pod="openshift-must-gather-9plmq/must-gather-q2rl2"
Nov 01 13:30:22.381341 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.381266    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9plmq/must-gather-q2rl2"
Nov 01 13:30:22.753487 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.753458    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9plmq/must-gather-q2rl2"]
Nov 01 13:30:22.760285 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:30:22.760255    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod782401f2_3cb5_4799_be6c_92e75186b676.slice/crio-33566782af7199ab3b12af4cd934bbba14c8b25570ae9ada48cbf40e164de417 WatchSource:0}: Error finding container 33566782af7199ab3b12af4cd934bbba14c8b25570ae9ada48cbf40e164de417: Status 404 returned error can't find the container with id 33566782af7199ab3b12af4cd934bbba14c8b25570ae9ada48cbf40e164de417
Nov 01 13:30:22.762138 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:22.762121    5725 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 13:30:23.133928 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:23.133840    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9plmq/must-gather-q2rl2" event={"ID":"782401f2-3cb5-4799-be6c-92e75186b676","Type":"ContainerStarted","Data":"33566782af7199ab3b12af4cd934bbba14c8b25570ae9ada48cbf40e164de417"}
Nov 01 13:30:25.150397 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:25.150361    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9plmq/must-gather-q2rl2" event={"ID":"782401f2-3cb5-4799-be6c-92e75186b676","Type":"ContainerStarted","Data":"172451741ce159798ad03a09df9cf277c7bb4565495a39ef41479ce8950ad76a"}
Nov 01 13:30:25.150397 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:25.150404    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9plmq/must-gather-q2rl2" event={"ID":"782401f2-3cb5-4799-be6c-92e75186b676","Type":"ContainerStarted","Data":"9a0b9bf8cc973900cf2abb491131efd3131ba0ade594478b653b3901ead85d96"}
Nov 01 13:30:25.259674 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:25.259622    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9plmq/must-gather-q2rl2" podStartSLOduration=1.989860657 podStartE2EDuration="3.259605599s" totalImagesPullingTime="1.269744942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:30:25.256964322 +0000 UTC m=+4066.764155540" watchObservedRunningTime="2026-11-01 13:30:25.259605599 +0000 UTC m=+4066.766796808"
Nov 01 13:30:31.037345 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:30:31.037306    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:30:33.808407 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:33.808376    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v7lhr_370a3bfd-1661-4e44-b5d5-5f6b83db1f6c/global-pull-secret-syncer/0.log"
Nov 01 13:30:34.965260 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:34.965229    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fg5h6_b6cb3539-d27f-4954-9b4d-e730dfd5809d/connection-checker/0.log"
Nov 01 13:30:35.380094 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:35.380041    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rsltv_3f8c18aa-2a61-4ddb-b0cf-75c0b76e45bf/connection-checker/0.log"
Nov 01 13:30:35.825011 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:35.824981    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tsmbq_bad6344f-4c79-4140-abcc-2c4ae3d63049/connection-checker/0.log"
Nov 01 13:30:36.672559 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:36.672527    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bm5xm_e5291295-5fd1-4e6e-b52b-c940c088168d/konnectivity-agent/0.log"
Nov 01 13:30:37.473060 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:37.473017    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-76.us-west-2.compute.internal_d08d3cfd44df29731faafa69be6f8df6/haproxy/0.log"
Nov 01 13:30:42.382728 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:42.382666    5725 scope.go:122] "RemoveContainer" containerID="5de6bcad240b7c7e1bc24c100e539dc0a21b7e91993153a3b2b42905f1be5b4a"
Nov 01 13:30:42.628097 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:42.628042    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f9995dd6f-pppw8_a563ddad-4fab-4b9f-9e01-c325efed519a/authorino/0.log"
Nov 01 13:30:42.939856 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:42.939809    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5d2lp_0d67bd5f-fba2-46dd-a457-124bf4b71a5f/manager/0.log"
Nov 01 13:30:43.329172 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.329138    5725 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"]
Nov 01 13:30:43.337065 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.337023    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.353361 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.353312    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zvk2f_6483676f-1bb7-4013-8246-aa877be51bb5/manager/0.log"
Nov 01 13:30:43.385228 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.384115    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"]
Nov 01 13:30:43.457529 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.457490    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-proc\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.457727 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.457548    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-podres\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.457727 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.457639    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-lib-modules\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.457833 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.457750    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbbff\" (UniqueName: \"kubernetes.io/projected/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-kube-api-access-cbbff\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.457833 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.457804    5725 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-sys\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.558992 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.558945    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbff\" (UniqueName: \"kubernetes.io/projected/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-kube-api-access-cbbff\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.559196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.559004    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-sys\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.559196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.559078    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-proc\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.559196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.559120    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-podres\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.559196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.559150    5725 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-lib-modules\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.559196 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.559161    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-sys\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.559406 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.559229    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-proc\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.559406 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.559238    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-lib-modules\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.559406 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.559267    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-podres\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.612498 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.612431    5725 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbff\" (UniqueName: \"kubernetes.io/projected/ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b-kube-api-access-cbbff\") pod \"perf-node-gather-daemonset-wg7ks\" (UID: \"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:43.660000 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:43.659973    5725 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:44.034083 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:44.034034    5725 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"]
Nov 01 13:30:44.041953 ip-10-0-138-76 kubenswrapper[5725]: W1101 13:30:44.041919    5725 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podee0ccd2c_8edb_4fc6_877d_6a51b392ca5b.slice/crio-7542aa4ddbb7c24b219dfb221a33c8e565cc1459438bd5ccb74aee1c9a37c305 WatchSource:0}: Error finding container 7542aa4ddbb7c24b219dfb221a33c8e565cc1459438bd5ccb74aee1c9a37c305: Status 404 returned error can't find the container with id 7542aa4ddbb7c24b219dfb221a33c8e565cc1459438bd5ccb74aee1c9a37c305
Nov 01 13:30:44.201339 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:44.201304    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bx6xs_838325b1-57bd-4388-a34f-e524c094fcc1/manager/0.log"
Nov 01 13:30:44.348776 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:44.348672    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks" event={"ID":"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b","Type":"ContainerStarted","Data":"e097a7944228400b7e85d11d77c835ef3aed3cb8ae22a5a29705b124c4efa321"}
Nov 01 13:30:44.348776 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:44.348739    5725 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks" event={"ID":"ee0ccd2c-8edb-4fc6-877d-6a51b392ca5b","Type":"ContainerStarted","Data":"7542aa4ddbb7c24b219dfb221a33c8e565cc1459438bd5ccb74aee1c9a37c305"}
Nov 01 13:30:44.348776 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:44.348768    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:44.455500 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:44.455445    5725 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks" podStartSLOduration=1.45543141 podStartE2EDuration="1.45543141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 13:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:30:44.454132668 +0000 UTC m=+4085.961323904" watchObservedRunningTime="2026-11-01 13:30:44.45543141 +0000 UTC m=+4085.962622627"
Nov 01 13:30:45.031370 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:45.031335    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4nhzf_73be14a6-2a72-424c-9fc9-4f5cc6fd9e53/manager/0.log"
Nov 01 13:30:50.369167 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:50.369136    5725 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-wg7ks"
Nov 01 13:30:50.570706 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:50.570665    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-jbpml_3758a7d1-ccc2-4322-bd01-e742e5c11ff4/cluster-monitoring-operator/0.log"
Nov 01 13:30:53.003904 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:53.003873    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4pdcm_5173ae11-40ca-47fe-a727-e39b21196422/node-exporter/0.log"
Nov 01 13:30:53.408722 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:53.408646    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4pdcm_5173ae11-40ca-47fe-a727-e39b21196422/kube-rbac-proxy/0.log"
Nov 01 13:30:53.816676 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:30:53.816645    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4pdcm_5173ae11-40ca-47fe-a727-e39b21196422/init-textfile/0.log"
Nov 01 13:31:01.026222 ip-10-0-138-76 kubenswrapper[5725]: E1101 13:31:01.026181    5725 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9xgxt" containerName="extractor"
Nov 01 13:31:06.032812 ip-10-0-138-76 kubenswrapper[5725]: I1101 13:31:06.032780    5725 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9clm8lm_7774276a-5194-4bbc-bb2b-7eb93e7fa6c2/istio-proxy/0.log"